2021-07-26T05:54:59.202 INFO:root:teuthology version: 1.1.0-b6f3efab 2021-07-26T05:54:59.204 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name kchai-2021-07-26_05:46:54-rados-wip-kefu-testing-2021-07-26-1205-distro-basic-smithi --archive /home/teuthworker/archive/kchai-2021-07-26_05:46:54-rados-wip-kefu-testing-2021-07-26-1205-distro-basic-smithi/6292742 --owner scheduled_kchai@teuthology --description rados/standalone/{supported-random-distro$/{centos_8} workloads/scrub} -- /home/teuthworker/archive/kchai-2021-07-26_05:46:54-rados-wip-kefu-testing-2021-07-26-1205-distro-basic-smithi/6292742/orig.config.yaml 2021-07-26T05:54:59.228 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-07-26T05:54:59.270 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-07-26_05:46:54-rados-wip-kefu-testing-2021-07-26-1205-distro-basic-smithi/6292742 branch: wip-kefu-testing-2021-07-26-1205 description: rados/standalone/{supported-random-distro$/{centos_8} workloads/scrub} email: null first_in_suite: false job_id: '6292742' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2021-07-26_05:46:54-rados-wip-kefu-testing-2021-07-26-1205-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.3' overrides: admin_socket: branch: wip-kefu-testing-2021-07-26-1205 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 197e564fdaee12596de1dc8f3253d1704ae66d5d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 197e564fdaee12596de1dc8f3253d1704ae66d5d selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-kefu-testing-2021-07-26-1205 sha1: 197e564fdaee12596de1dc8f3253d1704ae66d5d owner: scheduled_kchai@teuthology priority: 60 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 197e564fdaee12596de1dc8f3253d1704ae66d5d sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2021-07-26-1205 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_197e564fdaee12596de1dc8f3253d1704ae66d5d/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 197e564fdaee12596de1dc8f3253d1704ae66d5d targets: smithi071.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/Gdi8a4WpzLPRQkCAMloXrj1jdI2i7NIl3VC+HjMeYDzPEKsnoHcQSbD/Ff1DOM/McVLNx6oqc4vZT29Yyhht5MIiLZVZPxQ0sseCSWn765/ms+cBr++Iab7jdr5bH2hM4aKKq6ys3k+lPh0TEvHlNkqKWYDfILKtAWwdz78PO3HIGudFXtaGwNjQX5FesNL8FKr+kZYjSjXb1zDmjOY/EX23kWOnHK+eH/OHKSh5ScCz0i3LHdvzykIfEpvz4ObYVIxvqnBSxOOYgFaGRcbzXyYEMRaJucTb5wVi48URPm2OgeAbq00RJ01tI1O5eoXGyBiDn5U+LD0M1cW5AXhfHKJw5b/60TPLPluOsPRNCHBWWVxUOmp6Xy6mBNO5jkyPUKQkOt6msVpgav5ZsED7+fNCtQ9nIM8z0W9l1xG/KopufzOHstfo0j4Z+KSZnfsiHWaHEmMef4ASIbzmLOh3gW16lsAhXSVjZK6EC46yTnrnIjK7W+8t1YhFDiukIFs= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology_branch: master teuthology_sha1: b6f3efab84b70e18e86ec271a07564aeb4ed3bac timestamp: 2021-07-26_05:46:54 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.30581 2021-07-26T05:54:59.271 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_197e564fdaee12596de1dc8f3253d1704ae66d5d/qa; will attempt to use it 2021-07-26T05:54:59.271 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_197e564fdaee12596de1dc8f3253d1704ae66d5d/qa/tasks 2021-07-26T05:54:59.272 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-07-26T05:54:59.273 INFO:teuthology.task.internal:Checking packages... 2021-07-26T05:54:59.292 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '197e564fdaee12596de1dc8f3253d1704ae66d5d' 2021-07-26T05:54:59.292 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-07-26T05:54:59.293 INFO:teuthology.packaging:ref: None 2021-07-26T05:54:59.293 INFO:teuthology.packaging:tag: None 2021-07-26T05:54:59.293 INFO:teuthology.packaging:branch: wip-kefu-testing-2021-07-26-1205 2021-07-26T05:54:59.293 INFO:teuthology.packaging:sha1: 197e564fdaee12596de1dc8f3253d1704ae66d5d 2021-07-26T05:54:59.294 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-kefu-testing-2021-07-26-1205 2021-07-26T05:54:59.454 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-6358.g197e564f 2021-07-26T05:54:59.455 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-07-26T05:54:59.463 INFO:teuthology.task.internal:no buildpackages task found 2021-07-26T05:54:59.464 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-07-26T05:54:59.506 INFO:teuthology.task.internal:Saving configuration 2021-07-26T05:54:59.515 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-07-26T05:54:59.540 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-26T05:54:59.556 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-07-26 05:48:54.218826', 'description': '/home/teuthworker/archive/kchai-2021-07-26_05:46:54-rados-wip-kefu-testing-2021-07-26-1205-distro-basic-smithi/6292742', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDG+QeZI46yhGsuUTUznMWcbIswOJIhGeodF9RZ1rgbxQ48RytbnrwjKFiQvq1104MnovUw3mE00trVjr2bC4Uk=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi071.front.sepia.ceph.com'} 2021-07-26T05:54:59.556 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-07-26T05:54:59.573 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2021-07-26T05:54:59.574 INFO:teuthology.run_tasks:Running task console_log... 2021-07-26T05:54:59.629 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa8dfdc6d08>, signals=[15]) 2021-07-26T05:54:59.630 INFO:teuthology.run_tasks:Running task internal.connect... 2021-07-26T05:54:59.642 INFO:teuthology.task.internal:Opening connections... 2021-07-26T05:54:59.643 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2021-07-26T05:54:59.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-26T05:54:59.712 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-07-26T05:54:59.723 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2021-07-26T05:54:59.877 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2021-07-26T05:54:59.878 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2021-07-26T05:54:59.932 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Linux" 2021-07-26T05:54:59.932 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="8" 2021-07-26T05:54:59.932 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2021-07-26T05:54:59.932 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2021-07-26T05:54:59.933 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="8" 2021-07-26T05:54:59.933 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el8" 2021-07-26T05:54:59.933 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Linux 8" 2021-07-26T05:54:59.933 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2021-07-26T05:54:59.934 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-07-26T05:54:59.934 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2021-07-26T05:54:59.934 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-07-26T05:54:59.934 INFO:teuthology.orchestra.run.smithi071.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-07-26T05:54:59.934 INFO:teuthology.orchestra.run.smithi071.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-07-26T05:54:59.935 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2021-07-26T05:54:59.969 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-07-26T05:54:59.981 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-07-26T05:55:00.006 INFO:teuthology.task.internal:Checking for old test directory... 2021-07-26T05:55:00.007 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2021-07-26T05:55:00.020 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-07-26T05:55:00.040 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-07-26T05:55:00.040 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2021-07-26T05:55:00.086 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-07-26T05:55:00.114 INFO:teuthology.run_tasks:Running task kernel... 2021-07-26T05:55:00.139 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2021-07-26T05:55:00.139 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-07-26T05:55:00.140 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2021-07-26T05:55:00.154 INFO:teuthology.orchestra.run.smithi071.stdout:4.18.0-240.10.1.el8_3.x86_64 2021-07-26T05:55:00.154 INFO:teuthology.task.kernel:Running kernel on smithi071: 4.18.0-240.10.1.el8_3.x86_64 2021-07-26T05:55:00.154 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2021-07-26T05:55:03.834 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - AppStream 9.2 MB/s | 8.2 MB 00:00 2021-07-26T05:55:05.208 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - Base 39 MB/s | 4.5 MB 00:00 2021-07-26T05:55:05.913 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - Extras 145 kB/s | 9.8 kB 00:00 2021-07-26T05:55:09.001 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Linux 8 - AppStream 2.7 MB/s | 8.2 MB 00:03 2021-07-26T05:55:10.536 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Linux 8 - BaseOS 16 MB/s | 4.5 MB 00:00 2021-07-26T05:55:12.298 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Linux 8 - Extras 8.7 kB/s | 9.8 kB 00:01 2021-07-26T05:55:12.999 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 16 MB/s | 10 MB 00:00 2021-07-26T05:55:15.094 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2021-07-26T05:55:16.744 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-240.10.1.el8_3.x86_64 is already installed. 2021-07-26T05:55:16.744 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-07-26T05:55:17.001 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T05:55:17.017 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T05:55:17.018 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-07-26T05:55:17.018 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T05:55:17.018 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T05:55:17.018 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 4.18.0-305.10.2.el8_4 CentOS-Base 5.9 M 2021-07-26T05:55:17.019 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-07-26T05:55:17.019 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 4.18.0-305.10.2.el8_4 CentOS-Base 36 M 2021-07-26T05:55:17.019 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 4.18.0-305.10.2.el8_4 CentOS-Base 28 M 2021-07-26T05:55:17.019 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T05:55:17.019 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T05:55:17.020 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T05:55:17.020 INFO:teuthology.orchestra.run.smithi071.stdout:Install 3 Packages 2021-07-26T05:55:17.020 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T05:55:17.020 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 70 M 2021-07-26T05:55:17.020 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 88 M 2021-07-26T05:55:17.021 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T05:55:17.501 INFO:teuthology.orchestra.run.smithi071.stdout:(1/3): kernel-4.18.0-305.10.2.el8_4.x86_64.rpm 12 MB/s | 5.9 MB 00:00 2021-07-26T05:55:17.830 INFO:teuthology.orchestra.run.smithi071.stdout:(2/3): kernel-core-4.18.0-305.10.2.el8_4.x86_64 44 MB/s | 36 MB 00:00 2021-07-26T05:55:18.177 INFO:teuthology.orchestra.run.smithi071.stdout:(3/3): kernel-modules-4.18.0-305.10.2.el8_4.x86 24 MB/s | 28 MB 00:01 2021-07-26T05:55:18.181 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T05:55:18.182 INFO:teuthology.orchestra.run.smithi071.stdout:Total 60 MB/s | 70 MB 00:01 2021-07-26T05:55:18.293 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T05:55:18.611 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T05:55:18.611 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T05:55:19.868 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T05:55:19.884 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T05:55:22.359 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T05:55:22.842 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-4.18.0-305.10.2.el8_4.x86_64 1/3 2021-07-26T05:55:25.965 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-4.18.0-305.10.2.el8_4.x86_64 1/3 2021-07-26T05:55:26.274 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-4.18.0-305.10.2.el8_4.x86_64 2/3 2021-07-26T05:55:29.238 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-4.18.0-305.10.2.el8_4.x86_64 2/3 2021-07-26T05:55:29.477 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-4.18.0-305.10.2.el8_4.x86_64 3/3 2021-07-26T05:55:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-4.18.0-305.10.2.el8_4.x86_64 3/3 2021-07-26T05:55:50.053 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-4.18.0-305.10.2.el8_4.x86_64 3/3 2021-07-26T05:55:50.053 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-4.18.0-305.10.2.el8_4.x86_64 1/3 2021-07-26T05:55:50.054 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-4.18.0-305.10.2.el8_4.x86_64 2/3 2021-07-26T05:55:50.345 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-4.18.0-305.10.2.el8_4.x86_64 3/3 2021-07-26T05:55:50.345 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T05:55:50.718 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T05:55:50.718 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T05:55:50.718 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-4.18.0-305.10.2.el8_4.x86_64 2021-07-26T05:55:50.719 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core-4.18.0-305.10.2.el8_4.x86_64 2021-07-26T05:55:50.719 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-4.18.0-305.10.2.el8_4.x86_64 2021-07-26T05:55:50.719 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T05:55:50.719 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T05:55:50.788 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel --last | head -n 1 2021-07-26T05:55:50.922 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-4.18.0-305.10.2.el8_4.x86_64 Mon 26 Jul 2021 05:55:29 AM UTC 2021-07-26T05:55:50.924 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.10.2.el8_4.x86_64 2021-07-26T05:55:50.924 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-240.10.1.el8_3.x86_64 Expected: 4.18.0-305.10.2.el8_4.x86_64 2021-07-26T05:55:50.925 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-07-26T05:55:50.925 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-07-26T05:55:50.925 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2021-07-26T05:55:50.926 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2021-07-26T05:55:50.926 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel --last | head -n 1 2021-07-26T05:55:51.045 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-4.18.0-305.10.2.el8_4.x86_64 Mon 26 Jul 2021 05:55:29 AM UTC 2021-07-26T05:55:51.046 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.10.2.el8_4.x86_64 2021-07-26T05:55:51.047 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2021-07-26T05:55:51.077 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2021-07-26T05:55:51.077 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2021-07-26T05:55:51.078 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.02 2021-07-26T05:55:51.078 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 90.el8 2021-07-26T05:55:51.078 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2021-07-26T05:55:51.078 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Mon 01 Mar 2021 06:17:56 PM UTC 2021-07-26T05:55:51.079 INFO:teuthology.orchestra.run.smithi071.stdout:Group : System Environment/Base 2021-07-26T05:55:51.079 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 9197207 2021-07-26T05:55:51.079 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2021-07-26T05:55:51.079 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2021-07-26T05:55:51.079 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2021-07-26T05:55:51.080 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2021-07-26T05:55:51.080 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : kbuilder.bsys.centos.org 2021-07-26T05:55:51.080 INFO:teuthology.orchestra.run.smithi071.stdout:Relocations : (not relocatable) 2021-07-26T05:55:51.080 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : CentOS BuildSystem 2021-07-26T05:55:51.081 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2021-07-26T05:55:51.081 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2021-07-26T05:55:51.081 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2021-07-26T05:55:51.081 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2021-07-26T05:55:51.081 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T05:55:51.082 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-07-26T05:55:51.082 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2021-07-26T05:55:51.082 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2021-07-26T05:55:51.082 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2021-07-26T05:55:51.083 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T05:55:51.083 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2021-07-26T05:55:51.084 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-07-26T05:55:51.084 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 4.18.0-305.10.2.el8_4.x86_64 2021-07-26T05:55:51.084 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-07-26T05:55:52.868 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2021-07-26T05:55:55.263 INFO:teuthology.orchestra.run.smithi071.stderr:done 2021-07-26T05:55:55.265 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2021-07-26T05:55:55.294 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.6v1iUy2gy1 2021-07-26T05:55:55.294 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6v1iUy2gy1 2021-07-26T05:55:55.366 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.6v1iUy2gy1 2021-07-26T05:55:55.535 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.6v1iUy2gy1 is 6KB 2021-07-26T05:55:55.583 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.6v1iUy2gy1 2021-07-26T05:55:55.597 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2021-07-26T05:55:55.664 INFO:teuthology.orchestra.run.smithi071.stdout:631d84db90f7409fbdd390573ae89718-0-rescue.conf 2021-07-26T05:55:55.665 INFO:teuthology.orchestra.run.smithi071.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.10.1.el8_3.x86_64.conf 2021-07-26T05:55:55.665 INFO:teuthology.orchestra.run.smithi071.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.el8.x86_64.conf 2021-07-26T05:55:55.665 INFO:teuthology.orchestra.run.smithi071.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-305.10.2.el8_4.x86_64.conf 2021-07-26T05:55:55.666 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default 631d84db90f7409fbdd390573ae89718-4.18.0-305.10.2.el8_4.x86_64 2021-07-26T05:55:55.792 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2021-07-26T05:56:25.825 INFO:teuthology.misc:Re-opening connections... 2021-07-26T05:56:25.825 INFO:teuthology.misc:trying to connect to ubuntu@smithi071.front.sepia.ceph.com 2021-07-26T05:56:25.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-26T05:56:25.826 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-26T05:57:00.495 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2021-07-26T05:57:00.496 DEBUG:teuthology.misc:waited 34.67083477973938 2021-07-26T05:57:01.498 INFO:teuthology.misc:trying to connect to ubuntu@smithi071.front.sepia.ceph.com 2021-07-26T05:57:01.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-26T05:57:01.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-26T05:58:01.554 DEBUG:teuthology.orchestra.remote:timed out 2021-07-26T05:58:01.556 DEBUG:teuthology.misc:waited 95.73063731193542 2021-07-26T05:58:02.557 INFO:teuthology.misc:trying to connect to ubuntu@smithi071.front.sepia.ceph.com 2021-07-26T05:58:02.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-26T05:58:02.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-26T05:58:02.859 DEBUG:teuthology.orchestra.run.smithi071:> true 2021-07-26T05:58:03.158 DEBUG:teuthology.misc:waited 97.33249020576477 2021-07-26T05:58:04.159 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2021-07-26T05:58:04.160 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-305.10.2.el8_4.x86_64"... 2021-07-26T05:58:04.161 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2021-07-26T05:58:04.178 INFO:teuthology.orchestra.run.smithi071.stdout:4.18.0-305.10.2.el8_4.x86_64 2021-07-26T05:58:04.178 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-305.10.2.el8_4.x86_64 vs 4.18.0-305.10.2.el8_4.x86_64 2021-07-26T05:58:04.179 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-07-26T05:58:04.179 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-07-26T05:58:05.181 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-07-26T05:58:05.181 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-07-26T05:58:05.271 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2021-07-26T05:58:05.292 INFO:teuthology.run_tasks:Running task internal.base... 2021-07-26T05:58:05.305 INFO:teuthology.task.internal:Creating test directory... 2021-07-26T05:58:05.306 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-07-26T05:58:05.322 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-07-26T05:58:05.345 INFO:teuthology.run_tasks:Running task internal.archive... 2021-07-26T05:58:05.412 INFO:teuthology.task.internal:Creating archive directory... 2021-07-26T05:58:05.412 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-07-26T05:58:05.435 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-07-26T05:58:05.478 INFO:teuthology.task.internal:Enabling coredump saving... 2021-07-26T05:58:05.479 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-07-26T05:58:05.507 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T05:58:05.518 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T05:58:05.520 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-07-26T05:58:05.545 INFO:teuthology.task.internal:Configuring sudo... 2021-07-26T05:58:05.546 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-07-26T05:58:05.586 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-07-26T05:58:05.611 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-07-26T05:58:05.612 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-07-26T05:58:05.642 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-07-26T05:58:05.718 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-07-26T05:58:05.772 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-07-26T05:58:05.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-07-26T05:58:05.893 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-07-26T05:58:05.893 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-07-26T05:58:05.956 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2021-07-26T05:58:06.063 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-07-26T05:58:06.079 INFO:teuthology.run_tasks:Running task internal.timer... 2021-07-26T05:58:06.088 INFO:teuthology.task.internal:Starting timer... 2021-07-26T05:58:06.089 INFO:teuthology.run_tasks:Running task pcp... 2021-07-26T05:58:06.140 INFO:teuthology.run_tasks:Running task selinux... 2021-07-26T05:58:06.205 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-07-26T05:58:06.206 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2021-07-26T05:58:06.288 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: [ OK ] 2021-07-26T05:58:06.289 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-07-26T05:58:06.289 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2021-07-26T05:58:06.332 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2021-07-26T05:58:06.332 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi071.front.sepia.ceph.com': 'permissive'} 2021-07-26T05:58:06.333 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-07-26T05:58:06.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-26T05:58:06.397 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-07-26T05:58:06.397 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2021-07-26T05:58:06.462 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-07-26T05:58:06.476 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-07-26T05:58:06.478 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-07-26T05:58:06.496 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-07-26T05:58:06.498 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi071.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-07-26T05:58:06.839 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-07-26T05:58:07.646 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-07-26T05:58:10.486 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-07-26T05:58:10.510 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:10 +0000 (0:00:00.036) 0:00:00.036 *********** 2021-07-26T05:58:12.013 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi071.front.sepia.ceph.com] 2021-07-26T05:58:12.060 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:12 +0000 (0:00:01.551) 0:00:01.588 *********** 2021-07-26T05:58:12.496 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-07-26T05:58:12.498 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-07-26T05:58:12.518 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:12 +0000 (0:00:00.457) 0:00:02.046 *********** 2021-07-26T05:58:13.147 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** ok: [smithi071.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-07-26T05:58:13.169 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:13 +0000 (0:00:00.650) 0:00:02.697 *********** 2021-07-26T05:58:13.350 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-07-26T05:58:13.352 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:13.372 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:13 +0000 (0:00:00.203) 0:00:02.900 *********** 2021-07-26T05:58:13.805 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-07-26T05:58:13.807 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-26T05:58:13.931 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:13 +0000 (0:00:00.558) 0:00:03.459 *********** 2021-07-26T05:58:14.921 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-07-26T05:58:14.923 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627279093.97477-29612-212671761152784/source", "state": "file", "uid": 0} 2021-07-26T05:58:14.943 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:14 +0000 (0:00:01.011) 0:00:04.471 *********** 2021-07-26T05:58:15.382 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-07-26T05:58:15.384 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-07-26T05:58:15.386 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-07-26T05:58:15.447 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-07-26T05:58:15.448 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-07-26T05:58:15.465 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-07-26T05:58:15.466 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:15 +0000 (0:00:00.522) 0:00:04.994 *********** 2021-07-26T05:58:15.977 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] 2021-07-26T05:58:15.994 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** Monday 26 July 2021 05:58:15 +0000 (0:00:00.528) 0:00:05.522 *********** 2021-07-26T05:58:16.020 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-07-26T05:58:16.026 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-07-26T05:58:16.198 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:16 +0000 (0:00:00.204) 0:00:05.726 *********** 2021-07-26T05:58:16.819 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi071.front.sepia.ceph.com] 2021-07-26T05:58:16.868 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-07-26T05:58:16.870 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:16 +0000 (0:00:00.670) 0:00:06.396 *********** 2021-07-26T05:58:17.415 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-07-26T05:58:17.417 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.167665", "end": "2021-07-26 05:58:17.395599", "rc": 0, "start": "2021-07-26 05:58:17.227934", "stderr": "", "stderr_lines": [], "stdout": "fa3530b6c8007ac148c609947f616c9ce81a7370", "stdout_lines": ["fa3530b6c8007ac148c609947f616c9ce81a7370"]} 2021-07-26T05:58:17.438 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:17 +0000 (0:00:00.569) 0:00:06.966 *********** 2021-07-26T05:58:17.642 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-07-26T05:58:17.643 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013820", "end": "2021-07-26 05:58:17.623513", "failed_when_result": false, "rc": 0, "start": "2021-07-26 05:58:17.609693", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-07-26T05:58:17.668 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:17 +0000 (0:00:00.229) 0:00:07.196 *********** 2021-07-26T05:58:17.735 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-07-26T05:58:17.737 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:17.760 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:17 +0000 (0:00:00.092) 0:00:07.288 *********** 2021-07-26T05:58:17.829 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-07-26T05:58:17.831 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:17.852 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:17 +0000 (0:00:00.092) 0:00:07.380 *********** 2021-07-26T05:58:17.921 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-07-26T05:58:17.924 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:17.943 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:17 +0000 (0:00:00.090) 0:00:07.471 *********** 2021-07-26T05:58:18.013 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-07-26T05:58:18.015 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:18.035 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:18 +0000 (0:00:00.092) 0:00:07.563 *********** 2021-07-26T05:58:18.115 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-07-26T05:58:18.117 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:18.139 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:18 +0000 (0:00:00.103) 0:00:07.667 *********** 2021-07-26T05:58:18.363 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-07-26T05:58:18.365 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-26T05:58:18.388 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:18 +0000 (0:00:00.248) 0:00:07.916 *********** 2021-07-26T05:58:18.694 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-07-26T05:58:18.695 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-07-26T05:58:18.881 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-07-26T05:58:19.079 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-07-26T05:58:19.266 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-07-26T05:58:19.463 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-07-26T05:58:19.651 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-07-26T05:58:19.838 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-07-26T05:58:20.035 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-07-26T05:58:20.232 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-07-26T05:58:20.421 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-07-26T05:58:20.609 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-07-26T05:58:20.806 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-07-26T05:58:21.004 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-07-26T05:58:21.192 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-07-26T05:58:21.380 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-07-26T05:58:21.580 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-07-26T05:58:21.772 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-07-26T05:58:21.962 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-07-26T05:58:22.163 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-07-26T05:58:22.351 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-07-26T05:58:22.549 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-07-26T05:58:22.737 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-07-26T05:58:22.933 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-07-26T05:58:23.143 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-07-26T05:58:23.329 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-07-26T05:58:23.520 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-07-26T05:58:23.718 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-07-26T05:58:23.905 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-07-26T05:58:24.103 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-07-26T05:58:24.289 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-07-26T05:58:24.486 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-07-26T05:58:24.685 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-07-26T05:58:24.873 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-07-26T05:58:25.061 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-07-26T05:58:25.261 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-07-26T05:58:25.449 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-07-26T05:58:25.645 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-07-26T05:58:25.833 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-07-26T05:58:26.030 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-07-26T05:58:26.217 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-07-26T05:58:26.416 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-07-26T05:58:26.604 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-07-26T05:58:26.792 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-07-26T05:58:26.990 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-07-26T05:58:27.178 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-07-26T05:58:27.374 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-07-26T05:58:27.561 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-07-26T05:58:27.760 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-07-26T05:58:27.947 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-07-26T05:58:28.145 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-07-26T05:58:28.341 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-07-26T05:58:28.528 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-07-26T05:58:28.715 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-07-26T05:58:28.912 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-07-26T05:58:29.109 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-07-26T05:58:29.296 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-07-26T05:58:29.492 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-07-26T05:58:29.679 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-07-26T05:58:29.877 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-07-26T05:58:30.063 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-07-26T05:58:30.260 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-07-26T05:58:30.449 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-07-26T05:58:30.646 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-07-26T05:58:30.836 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-07-26T05:58:31.025 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-07-26T05:58:31.223 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-07-26T05:58:31.420 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-07-26T05:58:31.618 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-07-26T05:58:31.806 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-07-26T05:58:32.004 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-07-26T05:58:32.202 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-07-26T05:58:32.389 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-07-26T05:58:32.588 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-07-26T05:58:32.775 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-07-26T05:58:32.973 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-07-26T05:58:33.161 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-07-26T05:58:33.367 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-07-26T05:58:33.554 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-07-26T05:58:33.752 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-07-26T05:58:33.940 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-07-26T05:58:34.136 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-07-26T05:58:34.334 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-07-26T05:58:34.521 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-07-26T05:58:34.720 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-07-26T05:58:34.918 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-07-26T05:58:35.105 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-07-26T05:58:35.302 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-07-26T05:58:35.489 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-07-26T05:58:35.686 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-07-26T05:58:35.872 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-07-26T05:58:36.069 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-07-26T05:58:36.268 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-07-26T05:58:36.454 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-07-26T05:58:36.655 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-07-26T05:58:36.842 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-07-26T05:58:37.039 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-07-26T05:58:37.226 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-07-26T05:58:37.414 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-07-26T05:58:37.613 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-07-26T05:58:37.802 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-07-26T05:58:38.000 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-07-26T05:58:38.190 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-07-26T05:58:38.387 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-07-26T05:58:38.586 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-07-26T05:58:38.783 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-07-26T05:58:38.990 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-07-26T05:58:39.176 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-07-26T05:58:39.383 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-07-26T05:58:39.582 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-07-26T05:58:39.779 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-07-26T05:58:39.966 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-07-26T05:58:40.165 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-07-26T05:58:40.352 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-07-26T05:58:40.549 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-07-26T05:58:40.736 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-07-26T05:58:40.934 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-07-26T05:58:41.123 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-07-26T05:58:41.320 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-07-26T05:58:41.507 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-07-26T05:58:41.704 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-07-26T05:58:41.903 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-07-26T05:58:42.104 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-07-26T05:58:42.292 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-07-26T05:58:42.490 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-07-26T05:58:42.678 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-07-26T05:58:42.875 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-07-26T05:58:43.063 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-07-26T05:58:43.261 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-07-26T05:58:43.457 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-07-26T05:58:43.644 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-07-26T05:58:43.841 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-07-26T05:58:44.028 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-07-26T05:58:44.226 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-07-26T05:58:44.413 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-07-26T05:58:44.611 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-07-26T05:58:44.807 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-07-26T05:58:44.995 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-07-26T05:58:45.191 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-07-26T05:58:46.745 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "name": "dachary", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-07-26T05:58:48.603 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-07-26T05:58:50.143 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-07-26T05:58:51.591 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-07-26T05:58:53.103 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-07-26T05:58:54.661 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-07-26T05:58:56.212 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-07-26T05:58:57.713 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-07-26T05:58:59.297 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-07-26T05:59:00.872 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-07-26T05:59:02.484 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-07-26T05:59:04.061 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-07-26T05:59:05.626 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-07-26T05:59:07.207 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-07-26T05:59:08.821 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-07-26T05:59:08.960 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:08 +0000 (0:00:50.571) 0:00:58.488 *********** 2021-07-26T05:59:09.029 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-07-26T05:59:09.052 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:09 +0000 (0:00:00.091) 0:00:58.580 *********** 2021-07-26T05:59:09.142 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-07-26T05:59:09.144 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-07-26T05:59:09.146 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-07-26T05:59:09.147 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-07-26T05:59:09.149 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-07-26T05:59:09.150 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9 2021-07-26T05:59:09.152 INFO:teuthology.task.ansible.out:b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", " 2021-07-26T05:59:09.154 INFO:teuthology.task.ansible.out:ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 2021-07-26T05:59:09.155 INFO:teuthology.task.ansible.out:f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": 2021-07-26T05:59:09.157 INFO:teuthology.task.ansible.out:"alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon 2021-07-26T05:59:09.159 INFO:teuthology.task.ansible.out:", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"nam 2021-07-26T05:59:09.160 INFO:teuthology.task.ansible.out:e": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449 2021-07-26T05:59:09.162 INFO:teuthology.task.ansible.out:ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn 2021-07-26T05:59:09.164 INFO:teuthology.task.ansible.out:": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev 2021-07-26T05:59:09.164 INFO:teuthology.task.ansible.out: JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185 2021-07-26T05:59:09.166 INFO:teuthology.task.ansible.out:052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}]}, "changed": false} 2021-07-26T05:59:09.193 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:09 +0000 (0:00:00.140) 0:00:58.720 *********** 2021-07-26T05:59:10.247 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-07-26T05:59:10.249 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"after": "fa3530b6c8007ac148c609947f616c9ce81a7370", "attempts": 1, "before": "fa3530b6c8007ac148c609947f616c9ce81a7370", "changed": false, "remote_url_changed": false} 2021-07-26T05:59:10.269 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:10 +0000 (0:00:01.077) 0:00:59.797 *********** 2021-07-26T05:59:10.596 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-26T05:59:10.598 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-07-26T05:59:10.816 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-07-26T05:59:11.037 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-07-26T05:59:11.236 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-07-26T05:59:11.238 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-07-26T05:59:11.445 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-07-26T05:59:11.665 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-07-26T05:59:11.667 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-07-26T05:59:11.862 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-07-26T05:59:12.062 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-07-26T05:59:12.271 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-07-26T05:59:12.273 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:12.471 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-07-26T05:59:12.473 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-07-26T05:59:12.681 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-07-26T05:59:12.683 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-07-26T05:59:12.692 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:12.694 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:12.902 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-07-26T05:59:13.101 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-07-26T05:59:13.103 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-07-26T05:59:13.310 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-07-26T05:59:13.509 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-07-26T05:59:13.511 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-07-26T05:59:13.720 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-07-26T05:59:13.920 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-07-26T05:59:14.120 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-07-26T05:59:14.330 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-07-26T05:59:14.332 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-07-26T05:59:14.539 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-07-26T05:59:14.541 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-07-26T05:59:14.749 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-07-26T05:59:14.953 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-07-26T05:59:15.152 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-07-26T05:59:15.154 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-07-26T05:59:15.363 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-07-26T05:59:15.562 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-07-26T05:59:15.772 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-07-26T05:59:15.980 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-07-26T05:59:16.180 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-07-26T05:59:16.380 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-07-26T05:59:16.382 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-07-26T05:59:16.589 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-07-26T05:59:16.797 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-07-26T05:59:17.005 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-07-26T05:59:17.206 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-07-26T05:59:17.404 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-07-26T05:59:17.613 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-07-26T05:59:17.812 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-07-26T05:59:18.019 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-07-26T05:59:18.228 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-07-26T05:59:18.230 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-07-26T05:59:18.427 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-07-26T05:59:18.429 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-07-26T05:59:18.638 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-07-26T05:59:18.836 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-07-26T05:59:19.045 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-07-26T05:59:19.243 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtV 2021-07-26T05:59:19.245 INFO:teuthology.task.ansible.out:TFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-07-26T05:59:19.451 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-07-26T05:59:19.453 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-07-26T05:59:19.454 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-07-26T05:59:19.659 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-07-26T05:59:19.857 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-07-26T05:59:20.065 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-07-26T05:59:20.264 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-07-26T05:59:20.463 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-07-26T05:59:20.674 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-07-26T05:59:20.676 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-07-26T05:59:20.884 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-07-26T05:59:21.095 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-07-26T05:59:21.306 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-07-26T05:59:21.308 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-07-26T05:59:21.516 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-07-26T05:59:21.715 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-07-26T05:59:21.925 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-07-26T05:59:21.926 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-07-26T05:59:22.124 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-07-26T05:59:22.324 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-07-26T05:59:22.530 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-07-26T05:59:22.730 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-07-26T05:59:22.942 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-07-26T05:59:23.141 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-07-26T05:59:23.340 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-07-26T05:59:23.542 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-07-26T05:59:23.750 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-07-26T05:59:23.950 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-07-26T05:59:24.150 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-07-26T05:59:24.348 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-07-26T05:59:24.556 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-07-26T05:59:24.778 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-07-26T05:59:24.971 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-07-26T05:59:25.182 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-07-26T05:59:25.382 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-07-26T05:59:25.590 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-07-26T05:59:25.799 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-07-26T05:59:25.998 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-07-26T05:59:26.207 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-07-26T05:59:26.209 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-07-26T05:59:26.406 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-07-26T05:59:26.616 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-07-26T05:59:26.814 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-07-26T05:59:27.027 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-07-26T05:59:27.227 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-07-26T05:59:27.229 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-07-26T05:59:27.439 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-07-26T05:59:27.441 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-07-26T05:59:27.639 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-07-26T05:59:27.848 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-07-26T05:59:28.046 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-07-26T05:59:28.255 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-07-26T05:59:28.456 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-07-26T05:59:28.675 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-07-26T05:59:28.874 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-07-26T05:59:29.080 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-07-26T05:59:29.082 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-07-26T05:59:29.287 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-07-26T05:59:29.484 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-07-26T05:59:29.691 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-07-26T05:59:29.901 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-07-26T05:59:30.101 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-07-26T05:59:30.309 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-07-26T05:59:30.518 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-07-26T05:59:30.520 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-07-26T05:59:30.720 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-07-26T05:59:30.928 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-07-26T05:59:31.139 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-07-26T05:59:31.338 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-07-26T05:59:31.549 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-07-26T05:59:31.760 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-07-26T05:59:31.970 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-07-26T05:59:32.180 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-07-26T05:59:32.390 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-07-26T05:59:32.591 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-07-26T05:59:32.803 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-07-26T05:59:33.018 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-07-26T05:59:33.223 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-07-26T05:59:33.435 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-07-26T05:59:33.643 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-07-26T05:59:33.865 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-07-26T05:59:34.078 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-07-26T05:59:34.279 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-07-26T05:59:34.502 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-07-26T05:59:34.503 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-07-26T05:59:34.713 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-07-26T05:59:34.914 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-07-26T05:59:35.125 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-07-26T05:59:35.335 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-07-26T05:59:35.543 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-07-26T05:59:35.744 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-07-26T05:59:35.955 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-07-26T05:59:36.166 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-07-26T05:59:36.366 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-07-26T05:59:36.577 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-07-26T05:59:36.786 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-07-26T05:59:36.995 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-07-26T05:59:37.195 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-07-26T05:59:37.408 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-07-26T05:59:37.610 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-07-26T05:59:37.821 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-07-26T05:59:38.033 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-07-26T05:59:38.036 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-07-26T05:59:38.233 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-07-26T05:59:38.448 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-07-26T05:59:38.649 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-07-26T05:59:38.851 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-07-26T05:59:39.064 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-07-26T05:59:39.262 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-07-26T05:59:39.475 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-07-26T05:59:39.674 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-07-26T05:59:39.884 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-07-26T05:59:40.084 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-07-26T05:59:40.284 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-07-26T05:59:40.493 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICYMwYvykHtnVhhJwcKaiRh9oMpJkqvEA7OAbHPYOv6W matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-07-26T05:59:40.704 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-07-26T05:59:40.906 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-07-26T05:59:41.118 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-07-26T05:59:41.317 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-07-26T05:59:41.414 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:41 +0000 (0:00:31.144) 0:01:30.942 *********** 2021-07-26T05:59:41.504 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-26T05:59:41.506 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:41.516 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:41.517 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:41.527 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:41.549 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:41.552 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:41.564 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:41.573 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:41.795 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-07-26T05:59:41.806 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:41.817 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.017 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-07-26T05:59:42.215 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-07-26T05:59:42.226 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.228 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.238 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.248 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.250 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.260 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.271 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.272 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.282 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.283 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.294 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.295 INFO:teuthology.task.ansible.out: 2021-07-26T05:59:42.305 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.307 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.317 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.317 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.328 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.330 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.340 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.351 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.353 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.362 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.364 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.374 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.385 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.387 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.396 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.397 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.408 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.420 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.430 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.431 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.441 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.443 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.453 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.464 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.466 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.476 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.487 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.498 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.500 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.509 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.511 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.521 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.522 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.532 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.543 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.545 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.555 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.569 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.571 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.583 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.593 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.604 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.606 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.615 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.617 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.627 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.628 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.638 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.650 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.661 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.672 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.683 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.685 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.695 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.707 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.708 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.718 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.720 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.729 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.740 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.751 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.753 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.762 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.775 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.777 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.787 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.798 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.809 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.811 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.820 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.822 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.831 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.833 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.855 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.856 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.858 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.868 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.869 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.881 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.892 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.892 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.903 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.905 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.916 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.927 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.938 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.941 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.949 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.951 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.961 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.972 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.974 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.983 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.984 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.996 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.018 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.020 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.031 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.033 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.042 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.054 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.065 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.075 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.077 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.087 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.098 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.100 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.109 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.120 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.121 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.131 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.132 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.142 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.153 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.155 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.165 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.207 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:43 +0000 (0:00:01.793) 0:01:32.735 *********** 2021-07-26T05:59:43.276 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-07-26T05:59:43.278 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.299 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:43 +0000 (0:00:00.091) 0:01:32.827 *********** 2021-07-26T05:59:43.577 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-26T05:59:43.579 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-07-26T05:59:43.767 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-07-26T05:59:43.958 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-07-26T05:59:44.149 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-07-26T05:59:44.338 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-07-26T05:59:44.528 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-07-26T05:59:44.714 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-07-26T05:59:44.898 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-07-26T05:59:45.096 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-07-26T05:59:45.286 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-07-26T05:59:45.468 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-07-26T05:59:45.658 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-07-26T05:59:45.873 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-07-26T05:59:46.051 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-07-26T05:59:46.242 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-07-26T05:59:46.435 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-07-26T05:59:46.624 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-07-26T05:59:46.823 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-07-26T05:59:47.007 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-07-26T05:59:47.196 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-07-26T05:59:47.391 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-07-26T05:59:47.584 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-07-26T05:59:47.776 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-07-26T05:59:47.970 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-07-26T05:59:48.159 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-07-26T05:59:48.350 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-07-26T05:59:48.538 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-07-26T05:59:48.731 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-07-26T05:59:48.913 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-07-26T05:59:49.113 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-07-26T05:59:49.304 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-07-26T05:59:49.494 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-07-26T05:59:49.683 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-07-26T05:59:49.863 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-07-26T05:59:50.052 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-07-26T05:59:50.250 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-07-26T05:59:50.445 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-07-26T05:59:50.634 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-07-26T05:59:52.315 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jdillaman", "name": "jdillaman", "remove": false, "state": "absent"} 2021-07-26T05:59:52.504 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-07-26T05:59:52.712 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-07-26T05:59:52.902 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-07-26T05:59:53.092 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-07-26T05:59:53.280 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-07-26T05:59:53.469 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-07-26T05:59:53.660 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-07-26T05:59:53.851 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-07-26T05:59:54.039 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-07-26T05:59:54.239 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-07-26T05:59:54.281 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:54 +0000 (0:00:10.981) 0:01:43.809 *********** 2021-07-26T05:59:54.818 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-07-26T05:59:54.820 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "checksum": "6b66beacdb1c95b0ac86e1041f3914cfa0c50311", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "e34309bf71492baa6798d79247cca3d0", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627279194.3709736-12668-158160823998811/source", "state": "file", "uid": 0} 2021-07-26T05:59:54.840 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:54 +0000 (0:00:00.559) 0:01:44.368 *********** 2021-07-26T05:59:54.909 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-07-26T05:59:54.911 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => { "msg": "Host smithi071.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2021-07-26T05:59:54.932 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:54 +0000 (0:00:00.092) 0:01:44.460 *********** 2021-07-26T05:59:55.012 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-07-26T05:59:55.014 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-07-26T05:59:55.035 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:55 +0000 (0:00:00.102) 0:01:44.563 *********** 2021-07-26T05:59:55.106 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-07-26T05:59:55.108 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:55.128 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:55 +0000 (0:00:00.093) 0:01:44.656 *********** 2021-07-26T05:59:55.674 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-07-26T05:59:55.676 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.303945", "end": "2021-07-26 05:59:55.648104", "rc": 0, "start": "2021-07-26 05:59:55.344159", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-07-26T05:59:55.696 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:55 +0000 (0:00:00.567) 0:01:45.224 *********** 2021-07-26T05:59:55.764 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-07-26T05:59:55.766 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:55.788 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:55 +0000 (0:00:00.092) 0:01:45.316 *********** 2021-07-26T05:59:55.880 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:55.903 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:55 +0000 (0:00:00.114) 0:01:45.431 *********** 2021-07-26T05:59:55.972 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:55.995 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:55 +0000 (0:00:00.092) 0:01:45.523 *********** 2021-07-26T05:59:56.076 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-07-26T05:59:56.078 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:56.103 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:56 +0000 (0:00:00.107) 0:01:45.631 *********** 2021-07-26T05:59:56.175 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-07-26T05:59:56.177 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:56.201 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:56 +0000 (0:00:00.098) 0:01:45.729 *********** 2021-07-26T05:59:56.276 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-07-26T05:59:56.278 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:56.300 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:56 +0000 (0:00:00.098) 0:01:45.828 *********** 2021-07-26T05:59:56.379 INFO:teuthology.task.ansible.out: 2021-07-26T05:59:56.381 INFO:teuthology.task.ansible.out:TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi071.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-26T05:59:56.402 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:56 +0000 (0:00:00.102) 0:01:45.931 *********** 2021-07-26T05:59:56.473 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:56.496 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:56 +0000 (0:00:00.092) 0:01:46.023 *********** 2021-07-26T05:59:56.564 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-07-26T05:59:56.565 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:56.586 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:56 +0000 (0:00:00.090) 0:01:46.114 *********** 2021-07-26T05:59:56.667 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-07-26T05:59:56.669 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:56.703 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:56 +0000 (0:00:00.116) 0:01:46.231 *********** 2021-07-26T05:59:56.778 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-26T05:59:56.804 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:56 +0000 (0:00:00.100) 0:01:46.331 *********** 2021-07-26T05:59:56.883 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-07-26T05:59:56.884 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:56.909 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:56 +0000 (0:00:00.105) 0:01:46.437 *********** 2021-07-26T05:59:56.988 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-07-26T05:59:56.990 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-26T05:59:57.011 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.101) 0:01:46.539 *********** 2021-07-26T05:59:57.080 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-07-26T05:59:57.082 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:57.104 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.093) 0:01:46.633 *********** 2021-07-26T05:59:57.184 INFO:teuthology.task.ansible.out: 2021-07-26T05:59:57.186 INFO:teuthology.task.ansible.out:TASK [common : List CDN releases available to system] ************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:57.207 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.102) 0:01:46.735 *********** 2021-07-26T05:59:57.277 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:57.300 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.092) 0:01:46.828 *********** 2021-07-26T05:59:57.369 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-07-26T05:59:57.372 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:57.395 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.095) 0:01:46.923 *********** 2021-07-26T05:59:57.465 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-07-26T05:59:57.466 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-26T05:59:57.488 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.093) 0:01:47.016 *********** 2021-07-26T05:59:57.568 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-07-26T05:59:57.570 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:57.590 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.102) 0:01:47.118 *********** 2021-07-26T05:59:57.661 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:57.684 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.093) 0:01:47.212 *********** 2021-07-26T05:59:57.752 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-07-26T05:59:57.754 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:57.776 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.091) 0:01:47.304 *********** 2021-07-26T05:59:57.856 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-07-26T05:59:57.858 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:57.879 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.103) 0:01:47.408 *********** 2021-07-26T05:59:57.953 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-07-26T05:59:57.955 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:57.975 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.095) 0:01:47.503 *********** 2021-07-26T05:59:58.045 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:58.075 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:58 +0000 (0:00:00.096) 0:01:47.600 *********** 2021-07-26T05:59:58.151 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-26T05:59:58.175 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:58 +0000 (0:00:00.103) 0:01:47.703 *********** 2021-07-26T05:59:58.247 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-07-26T05:59:58.249 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:58.269 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:58 +0000 (0:00:00.094) 0:01:47.798 *********** 2021-07-26T05:59:58.340 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:58.361 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:58 +0000 (0:00:00.091) 0:01:47.889 *********** 2021-07-26T05:59:58.624 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-26T05:59:58.648 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:58 +0000 (0:00:00.286) 0:01:48.176 *********** 2021-07-26T05:59:59.491 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-07-26T05:59:59.493 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-07-26T05:59:59.933 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-07-26T05:59:59.958 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:59 +0000 (0:00:01.310) 0:01:49.486 *********** 2021-07-26T06:00:00.474 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-07-26T06:00:00.476 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-07-26T06:00:00.938 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-07-26T06:00:00.962 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:00 +0000 (0:00:01.003) 0:01:50.490 *********** 2021-07-26T06:00:01.042 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.065 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.103) 0:01:50.593 *********** 2021-07-26T06:00:01.135 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.157 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.092) 0:01:50.686 *********** 2021-07-26T06:00:01.226 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-07-26T06:00:01.228 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.249 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.091) 0:01:50.777 *********** 2021-07-26T06:00:01.318 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-07-26T06:00:01.320 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.340 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.090) 0:01:50.868 *********** 2021-07-26T06:00:01.410 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-07-26T06:00:01.412 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.422 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.425 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.451 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.111) 0:01:50.979 *********** 2021-07-26T06:00:01.521 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-07-26T06:00:01.523 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.544 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.092) 0:01:51.072 *********** 2021-07-26T06:00:01.612 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.634 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.090) 0:01:51.162 *********** 2021-07-26T06:00:01.703 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.727 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.092) 0:01:51.254 *********** 2021-07-26T06:00:01.794 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-07-26T06:00:01.796 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.817 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.090) 0:01:51.345 *********** 2021-07-26T06:00:01.893 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.913 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.096) 0:01:51.441 *********** 2021-07-26T06:00:02.515 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-07-26T06:00:02.517 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.416255", "end": "2021-07-26 06:00:02.500360", "rc": 0, "start": "2021-07-26 06:00:02.084105", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:00:02.538 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:02 +0000 (0:00:00.624) 0:01:52.066 *********** 2021-07-26T06:00:07.302 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-07-26T06:00:07.304 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-26T06:00:07.325 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:07 +0000 (0:00:04.786) 0:01:56.853 *********** 2021-07-26T06:00:07.396 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:07.420 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:07 +0000 (0:00:00.094) 0:01:56.947 *********** 2021-07-26T06:00:07.490 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-07-26T06:00:07.493 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:07.522 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:07 +0000 (0:00:00.102) 0:01:57.050 *********** 2021-07-26T06:00:07.603 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:07.626 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:07 +0000 (0:00:00.103) 0:01:57.154 *********** 2021-07-26T06:00:07.696 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-07-26T06:00:07.703 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:07.718 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:07 +0000 (0:00:00.091) 0:01:57.246 *********** 2021-07-26T06:00:08.204 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-07-26T06:00:08.206 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-07-26T06:00:08.226 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:08 +0000 (0:00:00.508) 0:01:57.754 *********** 2021-07-26T06:00:09.756 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-07-26T06:00:09.758 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-26T06:00:09.778 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:09 +0000 (0:00:01.552) 0:01:59.306 *********** 2021-07-26T06:00:10.303 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-07-26T06:00:10.305 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-07-26T06:00:10.733 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-07-26T06:00:11.153 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-07-26T06:00:11.177 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:11 +0000 (0:00:01.398) 0:02:00.705 *********** 2021-07-26T06:00:11.365 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-07-26T06:00:11.366 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-26T06:00:11.387 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:11 +0000 (0:00:00.210) 0:02:00.915 *********** 2021-07-26T06:00:11.874 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-07-26T06:00:11.876 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-07-26T06:00:12.287 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-07-26T06:00:12.756 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627279212.307894-15718-94763337384851/source", "state": "file", "uid": 0} 2021-07-26T06:00:12.779 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:12 +0000 (0:00:01.391) 0:02:02.307 *********** 2021-07-26T06:00:12.847 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-07-26T06:00:12.849 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:12.870 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:12 +0000 (0:00:00.090) 0:02:02.398 *********** 2021-07-26T06:00:12.951 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-07-26T06:00:12.953 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-07-26T06:00:12.973 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:12 +0000 (0:00:00.103) 0:02:02.501 *********** 2021-07-26T06:00:13.208 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-07-26T06:00:13.209 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-07-26T06:00:13.229 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:13 +0000 (0:00:00.256) 0:02:02.758 *********** 2021-07-26T06:00:13.319 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-07-26T06:00:13.321 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-26T06:00:13.330 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-26T06:00:13.352 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-26T06:00:13.364 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-26T06:00:13.387 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:13 +0000 (0:00:00.157) 0:02:02.915 *********** 2021-07-26T06:00:19.037 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: nagios-plugins-2.3.3-5.el8.x86_64", "Installed: nrpe-4.0.3-6.el8.x86_64", "Installed: nagios-plugins-load-2.3.3-5.el8.x86_64", "Installed: nagios-common-4.4.6-4.el8.x86_64", "Removed: nagios-common-4.4.5-1.el8.x86_64", "Removed: nagios-plugins-2.3.3-4.el8.x86_64", "Removed: nagios-plugins-load-2.3.3-4.el8.x86_64", "Removed: nrpe-4.0.3-1.el8.x86_64"]} 2021-07-26T06:00:19.060 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:19 +0000 (0:00:05.673) 0:02:08.588 *********** 2021-07-26T06:00:19.142 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-07-26T06:00:19.144 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:19.164 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:19 +0000 (0:00:00.104) 0:02:08.692 *********** 2021-07-26T06:00:19.669 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-07-26T06:00:19.671 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-07-26T06:00:19.692 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:19 +0000 (0:00:00.527) 0:02:09.220 *********** 2021-07-26T06:00:20.311 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-07-26T06:00:20.312 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-07-26T06:00:20.334 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:20 +0000 (0:00:00.642) 0:02:09.862 *********** 2021-07-26T06:00:20.414 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-07-26T06:00:20.416 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-07-26T06:00:20.425 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-07-26T06:00:20.448 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:20 +0000 (0:00:00.113) 0:02:09.976 *********** 2021-07-26T06:00:20.682 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-07-26T06:00:20.684 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-26T06:00:20.704 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:20 +0000 (0:00:00.256) 0:02:10.232 *********** 2021-07-26T06:00:20.982 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-07-26T06:00:20.984 INFO:teuthology.task.ansible.out:fatal: [smithi071.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-07-26T06:00:21.006 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:20 +0000 (0:00:00.301) 0:02:10.534 *********** 2021-07-26T06:00:21.074 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-07-26T06:00:21.076 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:21.097 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:21 +0000 (0:00:00.091) 0:02:10.625 *********** 2021-07-26T06:00:21.594 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-07-26T06:00:21.617 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:21 +0000 (0:00:00.519) 0:02:11.145 *********** 2021-07-26T06:00:22.390 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-07-26T06:00:22.392 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "15771", "ExecMainStartTimestamp": "Mon 2021-07-26 06:00:17 UTC", "ExecMainStartTimestampMonotonic": "179851729", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "984", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity" 2021-07-26T06:00:22.394 INFO:teuthology.task.ansible.out:, "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "15771", "MemoryAccounting": "yes", "MemoryCurrent": "733184", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "on-abort", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "Tim 2021-07-26T06:00:22.395 INFO:teuthology.task.ansible.out:eoutStopUSec": "1min", "TimerSlackNSec": "50000", "Type": "simple", "UID": "988", "UMask": "0022", "WatchdogTimestamp": "Mon 2021-07-26 06:00:17 UTC", "WatchdogTimestampMonotonic": "179851821", "WatchdogUSec": "0"}} 2021-07-26T06:00:22.414 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:22 +0000 (0:00:00.797) 0:02:11.942 *********** 2021-07-26T06:00:22.768 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-07-26T06:00:22.770 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001732", "end": "2021-07-26 06:00:22.749862", "rc": 0, "start": "2021-07-26 06:00:22.748130", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-07-26T06:00:22.790 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:22 +0000 (0:00:00.376) 0:02:12.318 *********** 2021-07-26T06:00:24.470 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-07-26T06:00:24.472 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-26T06:00:24.492 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:24 +0000 (0:00:01.702) 0:02:14.021 *********** 2021-07-26T06:01:01.298 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-07-26T06:01:01.301 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-67.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-67.el8.noarch", "Removed: selinux-policy-3.14.3-54.el8_3.2.noarch", "Removed: selinux-policy-targeted-3.14.3-54.el8_3.2.noarch"]} 2021-07-26T06:01:01.321 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:01 +0000 (0:00:36.828) 0:02:50.849 *********** 2021-07-26T06:01:02.111 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-07-26T06:01:02.112 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-07-26T06:01:02.134 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:02 +0000 (0:00:00.813) 0:02:51.662 *********** 2021-07-26T06:01:17.368 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-07-26T06:01:17.370 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:14.855304", "end": "2021-07-26 06:01:17.346814", "failed_when_result": false, "rc": 0, "start": "2021-07-26 06:01:02.491510", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-07-26T06:01:17.390 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:17 +0000 (0:00:15.255) 0:03:06.918 *********** 2021-07-26T06:01:17.881 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-07-26T06:01:17.882 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-07-26T06:01:17.905 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:17 +0000 (0:00:00.514) 0:03:07.433 *********** 2021-07-26T06:01:18.149 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-07-26T06:01:18.151 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.012323", "end": "2021-07-26 06:01:18.124213", "rc": 0, "start": "2021-07-26 06:01:18.111890", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:01:18.172 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:18 +0000 (0:00:00.267) 0:03:07.700 *********** 2021-07-26T06:01:18.445 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-07-26T06:01:18.446 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.035173", "end": "2021-07-26 06:01:18.421480", "rc": 0, "start": "2021-07-26 06:01:18.386307", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:01:18.476 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:18 +0000 (0:00:00.304) 0:03:08.004 *********** 2021-07-26T06:01:33.370 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-07-26T06:01:33.372 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:14.665960", "end": "2021-07-26 06:01:33.352702", "rc": 0, "start": "2021-07-26 06:01:18.686742", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:01:33.393 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:33 +0000 (0:00:14.916) 0:03:22.921 *********** 2021-07-26T06:01:33.633 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-07-26T06:01:33.635 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-07-26T06:01:33.655 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:33 +0000 (0:00:00.261) 0:03:23.183 *********** 2021-07-26T06:01:33.734 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-07-26T06:01:33.736 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-07-26T06:01:33.756 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:33 +0000 (0:00:00.101) 0:03:23.284 *********** 2021-07-26T06:01:33.836 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-07-26T06:01:33.838 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-07-26T06:01:33.859 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:33 +0000 (0:00:00.102) 0:03:23.387 *********** 2021-07-26T06:01:33.938 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-07-26T06:01:33.940 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuth 2021-07-26T06:01:33.942 INFO:teuthology.task.ansible.out:worker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-07-26T06:01:33.962 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:33 +0000 (0:00:00.103) 0:03:23.490 *********** 2021-07-26T06:01:34.044 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-07-26T06:01:34.046 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-07-26T06:01:34.068 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:34 +0000 (0:00:00.105) 0:03:23.596 *********** 2021-07-26T06:01:34.253 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-07-26T06:01:34.279 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:34 +0000 (0:00:00.211) 0:03:23.807 *********** 2021-07-26T06:01:34.513 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-07-26T06:01:34.515 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-07-26T06:01:34.535 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:34 +0000 (0:00:00.255) 0:03:24.063 *********** 2021-07-26T06:01:34.606 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-07-26T06:01:34.608 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:01:34.630 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:34 +0000 (0:00:00.094) 0:03:24.158 *********** 2021-07-26T06:01:34.699 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-07-26T06:01:34.757 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 26 July 2021 06:01:34 +0000 (0:00:00.091) 0:03:24.249 *********** 2021-07-26T06:01:34.980 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-07-26T06:01:34.982 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-07-26T06:01:35.004 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:34 +0000 (0:00:00.282) 0:03:24.532 *********** 2021-07-26T06:01:35.573 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-07-26T06:01:35.575 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-07-26T06:01:35.596 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:35 +0000 (0:00:00.591) 0:03:25.124 *********** 2021-07-26T06:01:35.677 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi071"}, "changed": false} 2021-07-26T06:01:35.701 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:35 +0000 (0:00:00.102) 0:03:25.227 *********** 2021-07-26T06:01:36.803 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-07-26T06:01:36.805 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi071.front.sepia.ceph.com", "ansible_hostname": "smithi071", "ansible_nodename": "smithi071"}, "changed": true, "name": "smithi071"} 2021-07-26T06:01:36.834 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:36 +0000 (0:00:01.135) 0:03:26.362 *********** 2021-07-26T06:01:37.369 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-07-26T06:01:37.398 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:37 +0000 (0:00:00.564) 0:03:26.926 *********** 2021-07-26T06:01:37.941 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-07-26T06:01:37.965 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:37 +0000 (0:00:00.565) 0:03:27.492 *********** 2021-07-26T06:01:38.546 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-07-26T06:01:38.548 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-07-26T06:01:38.572 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:38 +0000 (0:00:00.607) 0:03:28.099 *********** 2021-07-26T06:01:39.095 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-07-26T06:01:39.097 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-07-26T06:01:39.119 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:39 +0000 (0:00:00.548) 0:03:28.647 *********** 2021-07-26T06:01:39.358 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-07-26T06:01:39.381 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:39 +0000 (0:00:00.261) 0:03:28.909 *********** 2021-07-26T06:01:39.618 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-07-26T06:01:39.619 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-07-26T06:01:39.640 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:39 +0000 (0:00:00.258) 0:03:29.168 *********** 2021-07-26T06:01:39.902 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-07-26T06:01:39.904 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-07-26T06:01:39.928 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:39 +0000 (0:00:00.287) 0:03:29.456 *********** 2021-07-26T06:01:40.430 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-07-26T06:01:40.431 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-07-26T06:01:40.456 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:40 +0000 (0:00:00.528) 0:03:29.984 *********** 2021-07-26T06:01:40.950 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-07-26T06:01:40.952 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-07-26T06:01:40.975 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:40 +0000 (0:00:00.519) 0:03:30.503 *********** 2021-07-26T06:01:43.558 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-07-26T06:01:43.560 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.351597", "end": "2021-07-26 06:01:43.529852", "rc": 0, "start": "2021-07-26 06:01:41.178255", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:01:43.586 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:43 +0000 (0:00:02.610) 0:03:33.114 *********** 2021-07-26T06:01:43.668 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-07-26T06:01:43.670 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:01:43.690 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:43 +0000 (0:00:00.104) 0:03:33.218 *********** 2021-07-26T06:01:43.769 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T06:01:43.771 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:01:43.794 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:43 +0000 (0:00:00.104) 0:03:33.322 *********** 2021-07-26T06:01:43.876 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-07-26T06:01:43.878 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-07-26T06:01:43.899 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:43 +0000 (0:00:00.104) 0:03:33.427 *********** 2021-07-26T06:01:44.414 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-07-26T06:01:44.416 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-07-26T06:01:44.833 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-07-26T06:01:45.262 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-07-26T06:01:45.732 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "20107eac0ba9a84b318fa5f29b02a387051bff98", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "md5sum": "0b42b46569ffda9815a0a0455328d06a", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627279305.2918363-4286-144056453816993/source", "state": "file", "uid": 0} 2021-07-26T06:01:45.767 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:45 +0000 (0:00:01.868) 0:03:35.295 *********** 2021-07-26T06:01:46.285 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-07-26T06:01:46.287 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-07-26T06:01:46.309 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:46 +0000 (0:00:00.542) 0:03:35.837 *********** 2021-07-26T06:01:46.828 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-07-26T06:01:46.830 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-07-26T06:01:47.247 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-07-26T06:01:47.676 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-07-26T06:01:48.155 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "md5sum": "d96965513d5dae48e1f179ee52900b67", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627279307.7098067-4748-109339922135721/source", "state": "file", "uid": 0} 2021-07-26T06:01:48.185 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:48 +0000 (0:00:01.875) 0:03:37.713 *********** 2021-07-26T06:01:48.256 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-07-26T06:01:48.279 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:48 +0000 (0:00:00.094) 0:03:37.807 *********** 2021-07-26T06:01:48.351 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:01:48.383 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:48 +0000 (0:00:00.103) 0:03:37.911 *********** 2021-07-26T06:01:48.465 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-07-26T06:01:48.467 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-07-26T06:01:48.502 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:48 +0000 (0:00:00.118) 0:03:38.030 *********** 2021-07-26T06:01:49.068 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-07-26T06:01:49.070 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum module rather than running 'yum'. If you need to use command because yum is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.328673", "end": "2021-07-26 06:01:49.045362", "rc": 0, "start": "2021-07-26 06:01:48.716689", "stderr": "", "stderr_lines": [], "stdout": "64 files removed", "stdout_lines": ["64 files removed"]} 2021-07-26T06:01:49.091 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:49 +0000 (0:00:00.588) 0:03:38.619 *********** 2021-07-26T06:01:49.170 INFO:teuthology.task.ansible.out: 2021-07-26T06:01:49.172 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-07-26T06:01:49.194 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:49 +0000 (0:00:00.103) 0:03:38.722 *********** 2021-07-26T06:01:49.273 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-07-26T06:01:49.275 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:01:49.295 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:49 +0000 (0:00:00.101) 0:03:38.823 *********** 2021-07-26T06:01:49.374 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-07-26T06:01:49.376 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:01:49.398 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:49 +0000 (0:00:00.103) 0:03:38.926 *********** 2021-07-26T06:01:49.634 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-07-26T06:01:49.636 INFO:teuthology.task.ansible.out:fatal: [smithi071.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006404", "end": "2021-07-26 06:01:49.616876", "msg": "non-zero return code", "rc": 1, "start": "2021-07-26 06:01:49.610472", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-07-26T06:01:49.658 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:49 +0000 (0:00:00.259) 0:03:39.186 *********** 2021-07-26T06:01:49.758 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:01:49.781 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:01:49 +0000 (0:00:00.123) 0:03:39.309 *********** 2021-07-26T06:02:07.086 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-07-26T06:02:07.088 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64"]} 2021-07-26T06:02:07.108 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:07 +0000 (0:00:17.326) 0:03:56.636 *********** 2021-07-26T06:02:09.971 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-07-26T06:02:09.973 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-07-26T06:02:09.994 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:09 +0000 (0:00:02.886) 0:03:59.522 *********** 2021-07-26T06:02:19.526 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-26T06:02:19.528 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-plugins-core-4.0.18-4.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Installed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Installed: yum-utils-4.0.18-4.el8.noarch", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-common-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-core-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-img-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: qemu-kvm-common-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: qemu-kvm-core-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: python3-dnf-plugins-core-4.0.17-5.el8.noarch", "Removed: dnf-plugins-core-4.0.17-5.el8.noarch", "Removed: yum-utils-4.0.17-5.el8.noarch"]} 2021-07-26T06:02:19.548 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:19 +0000 (0:00:09.554) 0:04:09.077 *********** 2021-07-26T06:02:19.618 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-07-26T06:02:19.619 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:19.641 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:19 +0000 (0:00:00.092) 0:04:09.169 *********** 2021-07-26T06:02:21.398 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-07-26T06:02:21.400 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-26T06:02:21.421 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:21 +0000 (0:00:01.780) 0:04:10.950 *********** 2021-07-26T06:02:23.047 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-26T06:02:23.049 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-26T06:02:23.070 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:23 +0000 (0:00:01.648) 0:04:12.598 *********** 2021-07-26T06:02:23.420 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-07-26T06:02:23.422 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveState": "inactive", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity 2021-07-26T06:02:23.424 INFO:teuthology.task.ansible.out:", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-07-26T06:02:23.444 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:23 +0000 (0:00:00.373) 0:04:12.972 *********** 2021-07-26T06:02:23.776 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-07-26T06:02:23.777 INFO:teuthology.task.ansible.out:fatal: [smithi071.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-07-26T06:02:23.797 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:23 +0000 (0:00:00.353) 0:04:13.325 *********** 2021-07-26T06:02:24.501 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-07-26T06:02:24.503 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-07-26T06:02:24.524 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:24 +0000 (0:00:00.726) 0:04:14.052 *********** 2021-07-26T06:02:24.592 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-07-26T06:02:24.594 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:24.616 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:24 +0000 (0:00:00.092) 0:04:14.144 *********** 2021-07-26T06:02:24.684 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-07-26T06:02:24.686 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:24.706 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:24 +0000 (0:00:00.090) 0:04:14.235 *********** 2021-07-26T06:02:24.775 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-07-26T06:02:24.777 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:24.801 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:24 +0000 (0:00:00.094) 0:04:14.329 *********** 2021-07-26T06:02:24.872 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-07-26T06:02:24.874 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:24.905 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:24 +0000 (0:00:00.103) 0:04:14.433 *********** 2021-07-26T06:02:24.976 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:24.999 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:24 +0000 (0:00:00.094) 0:04:14.527 *********** 2021-07-26T06:02:25.067 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-07-26T06:02:25.068 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:25.090 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:25 +0000 (0:00:00.091) 0:04:14.618 *********** 2021-07-26T06:02:25.158 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-07-26T06:02:25.182 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:25 +0000 (0:00:00.091) 0:04:14.710 *********** 2021-07-26T06:02:25.250 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-07-26T06:02:25.252 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:25.275 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:25 +0000 (0:00:00.093) 0:04:14.803 *********** 2021-07-26T06:02:25.344 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-07-26T06:02:25.346 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:25.366 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:25 +0000 (0:00:00.090) 0:04:14.894 *********** 2021-07-26T06:02:25.445 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-07-26T06:02:25.447 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-07-26T06:02:25.469 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:25 +0000 (0:00:00.102) 0:04:14.997 *********** 2021-07-26T06:02:25.538 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-07-26T06:02:25.540 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:25.560 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:25 +0000 (0:00:00.091) 0:04:15.088 *********** 2021-07-26T06:02:25.628 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-07-26T06:02:25.650 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:25 +0000 (0:00:00.089) 0:04:15.178 *********** 2021-07-26T06:02:25.728 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-07-26T06:02:25.729 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:25.753 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:25 +0000 (0:00:00.103) 0:04:15.281 *********** 2021-07-26T06:02:25.822 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-07-26T06:02:25.824 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:25.845 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:25 +0000 (0:00:00.092) 0:04:15.373 *********** 2021-07-26T06:02:25.914 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-07-26T06:02:25.915 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:25.937 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:25 +0000 (0:00:00.091) 0:04:15.464 *********** 2021-07-26T06:02:26.006 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-26T06:02:26.008 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:26.028 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:26 +0000 (0:00:00.091) 0:04:15.556 *********** 2021-07-26T06:02:26.096 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-07-26T06:02:26.098 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:26.118 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:26 +0000 (0:00:00.089) 0:04:15.646 *********** 2021-07-26T06:02:26.185 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-26T06:02:26.187 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:26.208 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:26 +0000 (0:00:00.089) 0:04:15.736 *********** 2021-07-26T06:02:26.276 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-07-26T06:02:26.278 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:26.298 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:26 +0000 (0:00:00.090) 0:04:15.826 *********** 2021-07-26T06:02:26.367 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-07-26T06:02:26.369 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:26.391 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:26 +0000 (0:00:00.092) 0:04:15.919 *********** 2021-07-26T06:02:26.460 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-07-26T06:02:26.461 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:26.481 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:26 +0000 (0:00:00.090) 0:04:16.009 *********** 2021-07-26T06:02:26.550 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-07-26T06:02:26.551 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:26.572 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:26 +0000 (0:00:00.090) 0:04:16.100 *********** 2021-07-26T06:02:26.640 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-07-26T06:02:26.642 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:26.663 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:26 +0000 (0:00:00.091) 0:04:16.191 *********** 2021-07-26T06:02:26.735 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-07-26T06:02:26.736 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:26.759 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:26 +0000 (0:00:00.096) 0:04:16.287 *********** 2021-07-26T06:02:26.839 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-07-26T06:02:26.841 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:26.864 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:26 +0000 (0:00:00.104) 0:04:16.392 *********** 2021-07-26T06:02:26.932 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-07-26T06:02:26.934 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:26.955 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:26 +0000 (0:00:00.090) 0:04:16.483 *********** 2021-07-26T06:02:27.023 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-07-26T06:02:27.025 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:27.045 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:27 +0000 (0:00:00.090) 0:04:16.574 *********** 2021-07-26T06:02:27.113 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-07-26T06:02:27.114 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:27.134 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:27 +0000 (0:00:00.088) 0:04:16.662 *********** 2021-07-26T06:02:27.201 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-07-26T06:02:27.203 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:27.223 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:27 +0000 (0:00:00.089) 0:04:16.752 *********** 2021-07-26T06:02:27.292 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-07-26T06:02:27.294 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:27.315 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:27 +0000 (0:00:00.091) 0:04:16.843 *********** 2021-07-26T06:02:27.383 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-07-26T06:02:27.385 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:27.408 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:27 +0000 (0:00:00.091) 0:04:16.935 *********** 2021-07-26T06:02:27.475 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:27.497 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:27 +0000 (0:00:00.090) 0:04:17.025 *********** 2021-07-26T06:02:27.566 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-07-26T06:02:27.568 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:27.588 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:27 +0000 (0:00:00.090) 0:04:17.116 *********** 2021-07-26T06:02:27.657 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-07-26T06:02:27.659 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:27.679 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:27 +0000 (0:00:00.091) 0:04:17.208 *********** 2021-07-26T06:02:27.748 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-07-26T06:02:27.750 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:27.770 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:27 +0000 (0:00:00.090) 0:04:17.298 *********** 2021-07-26T06:02:27.838 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T06:02:27.840 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:27.860 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:27 +0000 (0:00:00.090) 0:04:17.388 *********** 2021-07-26T06:02:27.928 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-07-26T06:02:27.930 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:27.953 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:27 +0000 (0:00:00.093) 0:04:17.481 *********** 2021-07-26T06:02:28.021 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-26T06:02:28.022 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:28.042 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:28 +0000 (0:00:00.089) 0:04:17.571 *********** 2021-07-26T06:02:28.131 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-26T06:02:28.132 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:28.153 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:28 +0000 (0:00:00.110) 0:04:17.681 *********** 2021-07-26T06:02:28.220 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:28.242 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:28 +0000 (0:00:00.089) 0:04:17.770 *********** 2021-07-26T06:02:28.485 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-26T06:02:28.487 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-07-26T06:02:28.508 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:28 +0000 (0:00:00.266) 0:04:18.036 *********** 2021-07-26T06:02:28.765 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-26T06:02:28.767 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-26T06:02:28.790 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:28 +0000 (0:00:00.281) 0:04:18.318 *********** 2021-07-26T06:02:29.024 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-26T06:02:29.026 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-07-26T06:02:29.047 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:29 +0000 (0:00:00.256) 0:04:18.575 *********** 2021-07-26T06:02:29.115 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-07-26T06:02:29.116 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:29.136 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:29 +0000 (0:00:00.089) 0:04:18.664 *********** 2021-07-26T06:02:29.205 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-07-26T06:02:29.206 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:29.227 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:29 +0000 (0:00:00.090) 0:04:18.755 *********** 2021-07-26T06:02:29.294 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-07-26T06:02:29.296 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:29.316 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:29 +0000 (0:00:00.089) 0:04:18.844 *********** 2021-07-26T06:02:29.383 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-26T06:02:29.385 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:29.405 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:29 +0000 (0:00:00.088) 0:04:18.933 *********** 2021-07-26T06:02:29.473 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:29.497 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:29 +0000 (0:00:00.092) 0:04:19.025 *********** 2021-07-26T06:02:29.567 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:29.589 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:29 +0000 (0:00:00.091) 0:04:19.117 *********** 2021-07-26T06:02:29.657 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-26T06:02:29.659 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:29.680 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:29 +0000 (0:00:00.090) 0:04:19.208 *********** 2021-07-26T06:02:29.747 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:29.769 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:29 +0000 (0:00:00.089) 0:04:19.297 *********** 2021-07-26T06:02:29.836 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-26T06:02:29.838 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:29.864 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:29 +0000 (0:00:00.095) 0:04:19.393 *********** 2021-07-26T06:02:29.936 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-07-26T06:02:29.937 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:29.959 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:29 +0000 (0:00:00.094) 0:04:19.487 *********** 2021-07-26T06:02:30.031 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-07-26T06:02:30.033 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:30.054 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:30 +0000 (0:00:00.095) 0:04:19.582 *********** 2021-07-26T06:02:30.123 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-07-26T06:02:30.125 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:30.145 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:30 +0000 (0:00:00.090) 0:04:19.673 *********** 2021-07-26T06:02:30.213 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-07-26T06:02:30.215 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:30.235 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:30 +0000 (0:00:00.090) 0:04:19.763 *********** 2021-07-26T06:02:30.308 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-07-26T06:02:30.310 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:30.330 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:30 +0000 (0:00:00.095) 0:04:19.858 *********** 2021-07-26T06:02:30.409 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-07-26T06:02:30.411 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:30.432 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:30 +0000 (0:00:00.101) 0:04:19.960 *********** 2021-07-26T06:02:30.510 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-07-26T06:02:30.512 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:30.531 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:30 +0000 (0:00:00.099) 0:04:20.059 *********** 2021-07-26T06:02:30.599 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:30.623 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:30 +0000 (0:00:00.091) 0:04:20.151 *********** 2021-07-26T06:02:30.690 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-07-26T06:02:30.694 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:30.711 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:30 +0000 (0:00:00.088) 0:04:20.239 *********** 2021-07-26T06:02:30.777 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-07-26T06:02:30.779 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:30.799 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:30 +0000 (0:00:00.087) 0:04:20.327 *********** 2021-07-26T06:02:30.866 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:30.887 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:30 +0000 (0:00:00.088) 0:04:20.415 *********** 2021-07-26T06:02:30.944 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-07-26T06:02:30.946 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:30.967 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:30 +0000 (0:00:00.079) 0:04:20.495 *********** 2021-07-26T06:02:31.035 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-07-26T06:02:31.037 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:31.058 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:31 +0000 (0:00:00.090) 0:04:20.586 *********** 2021-07-26T06:02:31.127 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T06:02:31.129 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-07-26T06:02:31.149 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:31 +0000 (0:00:00.091) 0:04:20.677 *********** 2021-07-26T06:02:31.229 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-07-26T06:02:31.231 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-07-26T06:02:31.240 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-07-26T06:02:31.251 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-07-26T06:02:31.253 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-07-26T06:02:31.262 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-07-26T06:02:31.264 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-07-26T06:02:31.273 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-07-26T06:02:31.298 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:31 +0000 (0:00:00.148) 0:04:20.826 *********** 2021-07-26T06:02:31.520 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-07-26T06:02:31.522 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003276", "end": "2021-07-26 06:02:31.501646", "rc": 0, "start": "2021-07-26 06:02:31.498370", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/bin/teuthology-lock"]} 2021-07-26T06:02:31.545 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:31 +0000 (0:00:00.246) 0:04:21.073 *********** 2021-07-26T06:02:31.613 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:31.638 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:31 +0000 (0:00:00.093) 0:04:21.166 *********** 2021-07-26T06:02:31.706 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:31.730 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:31 +0000 (0:00:00.092) 0:04:21.258 *********** 2021-07-26T06:02:31.799 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-07-26T06:02:31.801 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:31.821 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:31 +0000 (0:00:00.090) 0:04:21.349 *********** 2021-07-26T06:02:31.899 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-07-26T06:02:31.901 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:31.924 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:31 +0000 (0:00:00.102) 0:04:21.452 *********** 2021-07-26T06:02:33.517 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-26T06:02:33.540 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:33 +0000 (0:00:01.615) 0:04:23.068 *********** 2021-07-26T06:02:33.619 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-07-26T06:02:33.621 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930251087872, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227112082, 'block_used': 12983026, 'inode_total': 61054976, 'inode_available': 60985735, 'inode_used': 69241, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227112082, "block_size": 4096, "block_total": 240095108, "block_used": 12983026, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985735, "inode_total": 61054976, "inode_used": 69241, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930251087872, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}} 2021-07-26T06:02:33.646 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:33 +0000 (0:00:00.106) 0:04:23.174 *********** 2021-07-26T06:02:33.879 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-07-26T06:02:33.881 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005088", "end": "2021-07-26 06:02:33.859154", "rc": 0, "start": "2021-07-26 06:02:33.854066", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:02:33.901 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:33 +0000 (0:00:00.254) 0:04:23.429 *********** 2021-07-26T06:02:33.979 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-07-26T06:02:33.981 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930251087872, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227112082, 'block_used': 12983026, 'inode_total': 61054976, 'inode_available': 60985735, 'inode_used': 69241, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227112082, "block_size": 4096, "block_total": 240095108, "block_used": 12983026, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985735, "inode_total": 61054976, "inode_used": 69241, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930251087872, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:34.001 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:33 +0000 (0:00:00.099) 0:04:23.529 *********** 2021-07-26T06:02:34.069 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-07-26T06:02:34.089 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:34 +0000 (0:00:00.088) 0:04:23.618 *********** 2021-07-26T06:02:35.975 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-26T06:02:35.977 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cHbe7jWZMTe6zxutmHnIhAWX6PjYHsxnYk'], 'uuids': ['806fe4b3-f3a5-41e2-a2a3-b063fbfe6e07'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.605237", "end": "2021-07-26 06:02:35.943456", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cHbe7jWZMTe6zxutmHnIhAWX6PjYHsxnYk"], "labels": [], "masters": [], "uuids": ["806fe4b3-f3a5-41e2-a2a3-b063fbfe6e07"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 06:02:34.338219", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-26T06:02:37.205 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-zYeOub-SGR5-tTwI-gFI0-EeHG-oMA3-jiLSP5', 'nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN', 'nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.043709", "end": "2021-07-26 06:02:37.174465", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-zYeOub-SGR5-tTwI-gFI0-EeHG-oMA3-jiLSP5", "nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN", "nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 06:02:36.130756", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-07-26T06:02:37.207 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-26T06:02:38.443 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cH0ptxeQaz31pfoW6E8Y0empzxFkJ0funy'], 'uuids': ['4795b537-aed6-40fd-a037-a7bd38cb1420'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.052525", "end": "2021-07-26 06:02:38.413118", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cH0ptxeQaz31pfoW6E8Y0empzxFkJ0funy"], "labels": [], "masters": [], "uuids": ["4795b537-aed6-40fd-a037-a7bd38cb1420"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 06:02:37.360593", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-26T06:02:39.689 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cHZzU0zlu0grJr2d7MQjV25u9FO11Pj5NM'], 'uuids': ['121578dc-2614-4c7c-b4b6-20ff047469a5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.060628", "end": "2021-07-26 06:02:39.663451", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cHZzU0zlu0grJr2d7MQjV25u9FO11Pj5NM"], "labels": [], "masters": [], "uuids": ["121578dc-2614-4c7c-b4b6-20ff047469a5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 06:02:38.602823", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-26T06:02:40.916 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cHL8huNrMRtY9umqD5axZauxPvU6XG3G8N'], 'uuids': ['6ed2d148-30f9-466d-85b3-7b8ccc04738c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.040652", "end": "2021-07-26 06:02:40.887040", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cHL8huNrMRtY9umqD5axZauxPvU6XG3G8N"], "labels": [], "masters": [], "uuids": ["6ed2d148-30f9-466d-85b3-7b8ccc04738c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 06:02:39.846388", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-26T06:02:40.928 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QJJR', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QJJR', 'scsi-35000c50091e19c4b', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR', 'wwn-0x5000c50091e19c4b'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QJJR', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e19c4b', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QJJR-part1', 'scsi-35000c50091e19c4b-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR-part1', 'wwn-0x5000c50091e19c4b-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QJJR", "scsi-35000c50091e19c4b", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR", "wwn-0x5000c50091e19c4b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-07-26T06:02:40.930 INFO:teuthology.task.ansible.out:Z1W5QJJR-part1", "scsi-35000c50091e19c4b-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR-part1", "wwn-0x5000c50091e19c4b-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QJJR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e19c4b"}}, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:42.176 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cH7aZFC9ml26eKAD3OI31CesXnEArKMpoa'], 'uuids': ['1c87954d-7f97-444e-83ea-ef67b24221a4'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.058186", "end": "2021-07-26 06:02:42.146893", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cH7aZFC9ml26eKAD3OI31CesXnEArKMpoa"], "labels": [], "masters": [], "uuids": ["1c87954d-7f97-444e-83ea-ef67b24221a4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 06:02:41.088707", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-26T06:02:42.202 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:42 +0000 (0:00:08.111) 0:04:31.730 *********** 2021-07-26T06:02:42.609 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-07-26T06:02:42.611 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cHbe7jWZMTe6zxutmHnIhAWX6PjYHsxnYk'], 'uuids': ['806fe4b3-f3a5-41e2-a2a3-b063fbfe6e07'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.146534", "end": "2021-07-26 06:02:42.580103", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cHbe7jWZMTe6zxutmHnIhAWX6PjYHsxnYk"], "labels": [], "masters": [], "uuids": ["806fe4b3-f3a5-41e2-a2a3-b063fbfe6e07"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 06:02:42.433569", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143828 s, 802 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143828 s, 802 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-26T06:02:42.818 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-zYeOub-SGR5-tTwI-gFI0-EeHG-oMA3-jiLSP5', 'nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN', 'nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.040463", "end": "2021-07-26 06:02:42.790522", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-zYeOub-SGR5-tTwI-gFI0-EeHG-oMA3-jiLSP5", "nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN", "nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 06:02:42.750059", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.037809 s, 3.1 GB/s", "stde 2021-07-26T06:02:42.821 INFO:teuthology.task.ansible.out:rr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.037809 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-26T06:02:43.132 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cH0ptxeQaz31pfoW6E8Y0empzxFkJ0funy'], 'uuids': ['4795b537-aed6-40fd-a037-a7bd38cb1420'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.145608", "end": "2021-07-26 06:02:43.112147", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cH0ptxeQaz31pfoW6E8Y0empzxFkJ0funy"], "labels": [], "masters": [], "uuids": ["4795b537-aed6-40fd-a037-a7bd38cb1420"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 06:02:42.966539", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142993 s, 807 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142993 s, 807 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-26T06:02:43.458 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cHZzU0zlu0grJr2d7MQjV25u9FO11Pj5NM'], 'uuids': ['121578dc-2614-4c7c-b4b6-20ff047469a5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.153687", "end": "2021-07-26 06:02:43.433227", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cHZzU0zlu0grJr2d7MQjV25u9FO11Pj5NM"], "labels": [], "masters": [], "uuids": ["121578dc-2614-4c7c-b4b6-20ff047469a5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 06:02:43.279540", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150988 s, 764 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150988 s, 764 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-26T06:02:43.794 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cHL8huNrMRtY9umqD5axZauxPvU6XG3G8N'], 'uuids': ['6ed2d148-30f9-466d-85b3-7b8ccc04738c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.157947", "end": "2021-07-26 06:02:43.761326", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cHL8huNrMRtY9umqD5axZauxPvU6XG3G8N"], "labels": [], "masters": [], "uuids": ["6ed2d148-30f9-466d-85b3-7b8ccc04738c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 06:02:43.603379", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.155328 s, 743 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.155328 s, 743 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-26T06:02:43.797 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QJJR', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QJJR', 'scsi-35000c50091e19c4b', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR', 'wwn-0x5000c50091e19c4b'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QJJR', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e19c4b', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QJJR-part1', 'scsi-35000c50091e19c4b-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR-part1', 'wwn-0x5000c50091e19c4b-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QJJR", "scsi-35000c50091e19c4b", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR", "wwn-0x5000c50091e19c4b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-07-26T06:02:43.798 INFO:teuthology.task.ansible.out:Z1W5QJJR-part1", "scsi-35000c50091e19c4b-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR-part1", "wwn-0x5000c50091e19c4b-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QJJR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e19c4b"}}, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:44.109 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cH7aZFC9ml26eKAD3OI31CesXnEArKMpoa'], 'uuids': ['1c87954d-7f97-444e-83ea-ef67b24221a4'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.145918", "end": "2021-07-26 06:02:44.082898", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-PXg0txWsopyasffBZVjfOoqcc5Vhe3cH7aZFC9ml26eKAD3OI31CesXnEArKMpoa"], "labels": [], "masters": [], "uuids": ["1c87954d-7f97-444e-83ea-ef67b24221a4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 06:02:43.936980", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143339 s, 805 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143339 s, 805 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-26T06:02:44.139 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:44 +0000 (0:00:01.937) 0:04:33.667 *********** 2021-07-26T06:02:44.415 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-07-26T06:02:44.417 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.051453", "end": "2021-07-26 06:02:44.348316", "rc": 0, "start": "2021-07-26 06:02:44.296863", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:02:44.437 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:44 +0000 (0:00:00.298) 0:04:33.965 *********** 2021-07-26T06:02:44.667 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-07-26T06:02:44.669 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.054934", "end": "2021-07-26 06:02:44.650268", "rc": 0, "start": "2021-07-26 06:02:44.595334", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:02:44.689 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:44 +0000 (0:00:00.252) 0:04:34.217 *********** 2021-07-26T06:02:44.758 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-07-26T06:02:44.817 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:44 +0000 (0:00:00.127) 0:04:34.345 *********** 2021-07-26T06:02:44.844 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-07-26T06:02:44.846 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:44.868 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:44 +0000 (0:00:00.051) 0:04:34.396 *********** 2021-07-26T06:02:44.895 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-07-26T06:02:44.897 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:44.917 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:44 +0000 (0:00:00.048) 0:04:34.445 *********** 2021-07-26T06:02:44.946 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:44.969 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:44 +0000 (0:00:00.051) 0:04:34.497 *********** 2021-07-26T06:02:44.996 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-07-26T06:02:44.997 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:45.017 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:45 +0000 (0:00:00.048) 0:04:34.545 *********** 2021-07-26T06:02:45.202 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-07-26T06:02:45.204 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-26T06:02:45.226 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:45 +0000 (0:00:00.208) 0:04:34.754 *********** 2021-07-26T06:02:45.264 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-07-26T06:02:45.266 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930251087872, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227112082, 'block_used': 12983026, 'inode_total': 61054976, 'inode_available': 60985735, 'inode_used': 69241, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227112082, "block_size": 4096, "block_total": 240095108, "block_used": 12983026, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985735, "inode_total": 61054976, "inode_used": 69241, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930251087872, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:45.289 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:45 +0000 (0:00:00.063) 0:04:34.817 *********** 2021-07-26T06:02:45.318 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-07-26T06:02:45.320 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:45.340 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:45 +0000 (0:00:00.051) 0:04:34.868 *********** 2021-07-26T06:02:45.368 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T06:02:45.369 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:45.391 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:45 +0000 (0:00:00.050) 0:04:34.919 *********** 2021-07-26T06:02:45.418 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T06:02:45.420 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:45.442 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:45 +0000 (0:00:00.051) 0:04:34.970 *********** 2021-07-26T06:02:45.470 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-07-26T06:02:45.472 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:45.504 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:45 +0000 (0:00:00.062) 0:04:35.032 *********** 2021-07-26T06:02:45.534 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-07-26T06:02:45.535 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:45.558 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:45 +0000 (0:00:00.053) 0:04:35.086 *********** 2021-07-26T06:02:45.597 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:45.620 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:45 +0000 (0:00:00.062) 0:04:35.149 *********** 2021-07-26T06:02:46.399 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-07-26T06:02:46.421 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:46 +0000 (0:00:00.800) 0:04:35.949 *********** 2021-07-26T06:02:47.104 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-07-26T06:02:47.106 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-26T06:02:47.437 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-26T06:02:47.789 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-26T06:02:48.132 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-26T06:02:48.480 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-26T06:02:48.507 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:48 +0000 (0:00:02.086) 0:04:38.035 *********** 2021-07-26T06:02:48.702 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-07-26T06:02:48.704 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-07-26T06:02:48.725 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:48 +0000 (0:00:00.217) 0:04:38.253 *********** 2021-07-26T06:02:48.928 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-07-26T06:02:48.930 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-26T06:02:49.113 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-26T06:02:49.290 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-26T06:02:49.456 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-26T06:02:49.468 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:49.493 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:49 +0000 (0:00:00.768) 0:04:39.021 *********** 2021-07-26T06:02:49.521 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-07-26T06:02:49.523 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:49.542 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:49 +0000 (0:00:00.048) 0:04:39.070 *********** 2021-07-26T06:02:49.568 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-07-26T06:02:49.570 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:49.590 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:49 +0000 (0:00:00.048) 0:04:39.119 *********** 2021-07-26T06:02:49.781 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-07-26T06:02:49.783 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-26T06:02:49.803 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:49 +0000 (0:00:00.212) 0:04:39.331 *********** 2021-07-26T06:02:49.840 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-07-26T06:02:49.841 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-07-26T06:02:49.865 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:49 +0000 (0:00:00.062) 0:04:39.393 *********** 2021-07-26T06:02:50.431 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-07-26T06:02:50.433 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true} 2021-07-26T06:02:50.455 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:50 +0000 (0:00:00.590) 0:04:39.984 *********** 2021-07-26T06:02:51.055 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-07-26T06:02:51.057 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-07-26T06:02:51.076 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.620) 0:04:40.604 *********** 2021-07-26T06:02:51.104 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-07-26T06:02:51.105 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.128 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.051) 0:04:40.656 *********** 2021-07-26T06:02:51.155 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-07-26T06:02:51.157 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.177 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.048) 0:04:40.705 *********** 2021-07-26T06:02:51.203 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-07-26T06:02:51.205 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.225 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.048) 0:04:40.753 *********** 2021-07-26T06:02:51.253 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-07-26T06:02:51.254 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.275 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.049) 0:04:40.803 *********** 2021-07-26T06:02:51.303 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-07-26T06:02:51.304 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.324 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.049) 0:04:40.852 *********** 2021-07-26T06:02:51.351 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-07-26T06:02:51.352 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.373 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.049) 0:04:40.901 *********** 2021-07-26T06:02:51.400 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-07-26T06:02:51.402 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.422 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.048) 0:04:40.950 *********** 2021-07-26T06:02:51.449 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-07-26T06:02:51.451 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.471 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.048) 0:04:40.999 *********** 2021-07-26T06:02:51.497 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.519 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.048) 0:04:41.047 *********** 2021-07-26T06:02:51.546 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-07-26T06:02:51.548 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.568 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.048) 0:04:41.096 *********** 2021-07-26T06:02:51.594 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-07-26T06:02:51.596 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.616 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.048) 0:04:41.144 *********** 2021-07-26T06:02:51.643 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-07-26T06:02:51.644 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.665 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.048) 0:04:41.193 *********** 2021-07-26T06:02:51.691 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-07-26T06:02:51.693 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.716 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.050) 0:04:41.244 *********** 2021-07-26T06:02:51.742 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-07-26T06:02:51.743 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.763 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.047) 0:04:41.292 *********** 2021-07-26T06:02:51.790 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-07-26T06:02:51.792 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:51.812 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:51 +0000 (0:00:00.048) 0:04:41.340 *********** 2021-07-26T06:02:52.272 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-07-26T06:02:52.274 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-07-26T06:02:52.295 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:52 +0000 (0:00:00.483) 0:04:41.823 *********** 2021-07-26T06:02:52.606 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-07-26T06:02:52.608 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "866", "ExecMainStartTimestamp": "Mon 2021-07-26 05:57:30 UTC", "ExecMainStartTimestampMonotonic": "12826844", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSI 2021-07-26T06:02:52.608 INFO:teuthology.task.ansible.out:ZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "866", "MemoryAccounting": "yes", "MemoryCurrent": "1826816", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "forking", "UID": "[not set]", "UMask": "0022", "WatchdogTimestamp": "Mon 2021-07-26 0 2021-07-26T06:02:52.610 INFO:teuthology.task.ansible.out:5:57:30 UTC", "WatchdogTimestampMonotonic": "12826866", "WatchdogUSec": "0"}} 2021-07-26T06:02:52.628 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:52 +0000 (0:00:00.333) 0:04:42.156 *********** 2021-07-26T06:02:52.654 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T06:02:52.656 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-07-26T06:02:52.675 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:52 +0000 (0:00:00.046) 0:04:42.203 *********** 2021-07-26T06:02:52.712 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T06:02:52.714 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-07-26T06:02:52.734 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:52 +0000 (0:00:00.058) 0:04:42.262 *********** 2021-07-26T06:02:52.761 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-07-26T06:02:52.763 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:52.783 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:52 +0000 (0:00:00.048) 0:04:42.311 *********** 2021-07-26T06:02:52.809 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:52.831 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:52 +0000 (0:00:00.048) 0:04:42.359 *********** 2021-07-26T06:02:52.857 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-07-26T06:02:52.859 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:52.879 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:52 +0000 (0:00:00.048) 0:04:42.407 *********** 2021-07-26T06:02:53.071 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-07-26T06:02:53.073 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-07-26T06:02:53.093 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:53 +0000 (0:00:00.213) 0:04:42.621 *********** 2021-07-26T06:02:53.565 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-07-26T06:02:53.567 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-07-26T06:02:53.587 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:53 +0000 (0:00:00.493) 0:04:43.115 *********** 2021-07-26T06:02:55.010 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-07-26T06:02:55.012 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-07-26T06:02:55.032 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:55 +0000 (0:00:01.445) 0:04:44.560 *********** 2021-07-26T06:02:55.058 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-07-26T06:02:55.060 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:55.079 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:55 +0000 (0:00:00.047) 0:04:44.607 *********** 2021-07-26T06:02:55.105 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-07-26T06:02:55.106 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-07-26T06:02:55.127 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:55 +0000 (0:00:00.047) 0:04:44.655 *********** 2021-07-26T06:02:55.302 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-07-26T06:02:55.304 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-07-26T06:02:55.325 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:55 +0000 (0:00:00.198) 0:04:44.853 *********** 2021-07-26T06:02:55.353 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-07-26T06:02:55.355 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:55.375 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:55 +0000 (0:00:00.050) 0:04:44.904 *********** 2021-07-26T06:02:55.402 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-07-26T06:02:55.404 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-07-26T06:02:55.424 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:55 +0000 (0:00:00.048) 0:04:44.952 *********** 2021-07-26T06:02:55.451 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-07-26T06:02:55.453 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:55.473 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:55 +0000 (0:00:00.049) 0:04:45.002 *********** 2021-07-26T06:02:55.656 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-07-26T06:02:55.658 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-07-26T06:02:55.679 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:55 +0000 (0:00:00.205) 0:04:45.207 *********** 2021-07-26T06:02:55.864 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-07-26T06:02:55.866 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-07-26T06:02:55.886 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:55 +0000 (0:00:00.207) 0:04:45.414 *********** 2021-07-26T06:02:55.913 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-07-26T06:02:55.915 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:55.934 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:55 +0000 (0:00:00.048) 0:04:45.463 *********** 2021-07-26T06:02:55.972 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-07-26T06:02:55.974 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-07-26T06:02:55.984 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-07-26T06:02:55.995 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-07-26T06:02:55.996 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-07-26T06:02:56.017 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:56 +0000 (0:00:00.082) 0:04:45.545 *********** 2021-07-26T06:02:56.056 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-07-26T06:02:56.058 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:56.069 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:56.070 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:56.092 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:56 +0000 (0:00:00.074) 0:04:45.620 *********** 2021-07-26T06:02:56.119 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T06:02:56.121 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:56.144 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:56 +0000 (0:00:00.051) 0:04:45.671 *********** 2021-07-26T06:02:56.171 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-07-26T06:02:56.173 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:56.202 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:56 +0000 (0:00:00.058) 0:04:45.730 *********** 2021-07-26T06:02:56.229 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-07-26T06:02:56.231 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:56.256 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:56 +0000 (0:00:00.053) 0:04:45.784 *********** 2021-07-26T06:02:56.284 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:56.307 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:56 +0000 (0:00:00.050) 0:04:45.835 *********** 2021-07-26T06:02:56.335 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:56.359 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:56 +0000 (0:00:00.052) 0:04:45.887 *********** 2021-07-26T06:02:56.387 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-07-26T06:02:56.389 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:56.417 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:56 +0000 (0:00:00.057) 0:04:45.945 *********** 2021-07-26T06:02:56.445 INFO:teuthology.task.ansible.out: 2021-07-26T06:02:56.446 INFO:teuthology.task.ansible.out:TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:56.470 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:56 +0000 (0:00:00.052) 0:04:45.997 *********** 2021-07-26T06:02:56.496 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-07-26T06:02:56.498 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:56.518 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:56 +0000 (0:00:00.048) 0:04:46.046 *********** 2021-07-26T06:02:56.723 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-07-26T06:02:56.725 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005024", "end": "2021-07-26 06:02:56.696194", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-07-26 06:02:56.691170", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:02:56.890 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003165", "end": "2021-07-26 06:02:56.865959", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-07-26 06:02:56.862794", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:02:57.056 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002842", "end": "2021-07-26 06:02:57.032168", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-07-26 06:02:57.029326", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-07-26T06:02:57.225 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003230", "end": "2021-07-26 06:02:57.197998", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-07-26 06:02:57.194768", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:02:57.246 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:57 +0000 (0:00:00.727) 0:04:46.774 *********** 2021-07-26T06:02:57.273 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-07-26T06:02:57.297 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:57 +0000 (0:00:00.050) 0:04:46.825 *********** 2021-07-26T06:02:57.334 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:57.402 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-07-26T06:02:57.419 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:57 +0000 (0:00:00.122) 0:04:46.947 *********** 2021-07-26T06:02:58.053 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi071.front.sepia.ceph.com] 2021-07-26T06:02:58.100 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:58 +0000 (0:00:00.681) 0:04:47.629 *********** 2021-07-26T06:02:58.127 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-26T06:02:58.129 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:02:58.149 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:58 +0000 (0:00:00.048) 0:04:47.677 *********** 2021-07-26T06:02:58.176 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-26T06:02:58.178 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-07-26T06:02:58.198 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:58 +0000 (0:00:00.049) 0:04:47.726 *********** 2021-07-26T06:02:58.245 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-07-26T06:02:58.247 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-07-26T06:02:58.270 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:02:58 +0000 (0:00:00.071) 0:04:47.798 *********** 2021-07-26T06:03:04.312 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-catatonit-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Installed: podman-docker-3.0.1-7.module_el8.4.0+830+8027e1c4.noarch", "Installed: podman-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64", "Removed: podman-catatonit-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64", "Removed: podman-docker-2.2.1-7.module_el8.3.0+699+d61d9c41.noarch"]} 2021-07-26T06:03:04.335 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:03:04 +0000 (0:00:06.065) 0:04:53.863 *********** 2021-07-26T06:03:04.362 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-07-26T06:03:04.384 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:03:04 +0000 (0:00:00.049) 0:04:53.912 *********** 2021-07-26T06:03:04.411 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-26T06:03:04.413 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:03:04.435 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:03:04 +0000 (0:00:00.050) 0:04:53.963 *********** 2021-07-26T06:03:04.651 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-07-26T06:03:04.653 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-26T06:03:04.842 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-26T06:03:04.868 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:03:04 +0000 (0:00:00.432) 0:04:54.396 *********** 2021-07-26T06:03:05.402 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-07-26T06:03:05.404 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627279384.9356558-15601-236601057385600/source", "state": "file", "uid": 0} 2021-07-26T06:03:05.914 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627279385.4448721-15601-123374370669199/source", "state": "file", "uid": 0} 2021-07-26T06:03:05.937 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:03:05 +0000 (0:00:01.069) 0:04:55.465 *********** 2021-07-26T06:03:07.556 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-26T06:03:07.579 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:03:07 +0000 (0:00:01.641) 0:04:57.107 *********** 2021-07-26T06:03:10.763 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-07-26T06:03:10.765 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-dehewqni-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-dehewqni-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-07-26T06:03:10.767 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-07-26T06:03:10.786 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:03:10 +0000 (0:00:03.207) 0:05:00.314 *********** 2021-07-26T06:03:10.824 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-07-26T06:03:10.825 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:03:10.851 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:03:10 +0000 (0:00:00.064) 0:05:00.379 *********** 2021-07-26T06:03:10.889 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-07-26T06:03:10.890 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:03:10.912 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:03:10 +0000 (0:00:00.060) 0:05:00.440 *********** 2021-07-26T06:03:11.274 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.170660", "end": "2021-07-26 06:03:11.254473", "rc": 0, "start": "2021-07-26 06:03:11.083813", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:03:11.295 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:03:11 +0000 (0:00:00.383) 0:05:00.824 *********** 2021-07-26T06:03:11.323 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-07-26T06:03:11.325 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:03:11.391 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-07-26T06:03:11.392 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-07-26T06:03:11.393 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-07-26T06:03:11.394 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-07-26T06:03:11.407 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-07-26T06:03:11.408 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:03:11 +0000 (0:00:00.111) 0:05:00.935 *********** 2021-07-26T06:03:12.041 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] 2021-07-26T06:03:12.057 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** Monday 26 July 2021 06:03:12 +0000 (0:00:00.650) 0:05:01.585 *********** 2021-07-26T06:03:12.264 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-07-26T06:03:12.266 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi071.front.sepia.ceph.com : ok=132 changed=47 unreachable=0 failed=0 skipped=188 rescued=0 ignored=3 2021-07-26T06:03:12.267 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:03:12 +0000 (0:00:00.209) 0:05:01.795 *********** =============================================================================== 2021-07-26T06:03:12.268 INFO:teuthology.task.ansible.out:Create all admin users with sudo access. ------------------------------- 50.57s 2021-07-26T06:03:12.268 INFO:teuthology.task.ansible.out:common : nrpe - Ensure SELinux policy is up to date -------------------- 36.83s users : Update authorized_keys using the keys repo --------------------- 31.14s testnode : Ensure ceph packages are not present. ----------------------- 17.33s common : nrpe - Remove SELinux policy package -------------------------- 15.26s common : nrpe - Load SELinux policy package ---------------------------- 14.92s Remove revoked users --------------------------------------------------- 10.98s testnode : Install packages --------------------------------------------- 9.55s testnode : Zap all non-root disks --------------------------------------- 8.11s container-host : Install container packages ----------------------------- 6.07s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 5.67s common : Install Kerberos Packages (RedHat) ----------------------------- 4.79s container-host : Install registries-conf-ctl ---------------------------- 3.21s testnode : Ensure ceph dependency packages are not present. ------------- 2.89s testnode : Rebuild rpmdb ------------------------------------------------ 2.61s testnode : Create logical volume(s) ------------------------------------- 2.09s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.94s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.88s testnode : Configure local mirrorlists ---------------------------------- 1.87s users : Update authorized_keys for each user with literal keys ---------- 1.79s 2021-07-26T06:03:12.660 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi071.front.sepia.ceph.com')] 2021-07-26T06:03:12.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-26T06:03:12.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-26T06:03:12.742 DEBUG:teuthology.orchestra.run.smithi071:> true 2021-07-26T06:03:12.822 INFO:teuthology.run_tasks:Running task clock... 2021-07-26T06:03:12.839 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-07-26T06:03:12.839 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-07-26T06:03:12.841 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-07-26T06:03:12.894 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-07-26T06:03:12.909 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-07-26T06:03:12.949 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2021-07-26T06:03:12.958 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2021-07-26T06:03:12.972 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-07-26T06:03:12.985 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-07-26T06:03:13.049 INFO:teuthology.orchestra.run.smithi071.stderr:bash: ntpq: command not found 2021-07-26T06:03:13.051 INFO:teuthology.orchestra.run.smithi071.stdout:210 Number of sources = 4 2021-07-26T06:03:13.051 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-07-26T06:03:13.052 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2021-07-26T06:03:13.052 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-26T06:03:13.052 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-26T06:03:13.052 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-26T06:03:13.052 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-26T06:03:13.053 INFO:teuthology.run_tasks:Running task install... 2021-07-26T06:03:13.063 DEBUG:teuthology.task.install:project ceph 2021-07-26T06:03:13.064 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '197e564fdaee12596de1dc8f3253d1704ae66d5d'}} 2021-07-26T06:03:13.064 DEBUG:teuthology.task.install:config {'sha1': '197e564fdaee12596de1dc8f3253d1704ae66d5d'} 2021-07-26T06:03:13.065 INFO:teuthology.task.install:Using flavor: basic 2021-07-26T06:03:13.071 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-07-26T06:03:13.071 INFO:teuthology.task.install:extra packages: [] 2021-07-26T06:03:13.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo lsb_release -is 2021-07-26T06:03:13.124 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS 2021-07-26T06:03:13.125 DEBUG:teuthology.misc:System to be installed: CentOS 2021-07-26T06:03:13.126 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '197e564fdaee12596de1dc8f3253d1704ae66d5d', 'tag': None, 'wait_for_package': False} 2021-07-26T06:03:13.127 DEBUG:teuthology.orchestra.run.smithi071:> sudo lsb_release -is 2021-07-26T06:03:13.198 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS 2021-07-26T06:03:13.199 DEBUG:teuthology.misc:System to be installed: CentOS 2021-07-26T06:03:13.200 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=197e564fdaee12596de1dc8f3253d1704ae66d5d 2021-07-26T06:03:13.365 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/centos/8/flavors/default/ 2021-07-26T06:03:13.365 INFO:teuthology.task.install.rpm:Package version is 17.0.0-6358.g197e564f 2021-07-26T06:03:13.509 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-07-26T06:03:13.510 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-07-26T06:03:13.510 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-07-26T06:03:13.535 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-07-26T06:03:13.536 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/197e564fdaee12596de1dc8f3253d1704ae66d5d/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-07-26T06:03:13.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-07-26T06:03:13.674 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-07-26T06:03:13.739 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2021-07-26T06:03:13.740 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2021-07-26T06:03:14.068 INFO:teuthology.orchestra.run.smithi071.stdout:80 files removed 2021-07-26T06:03:14.103 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2021-07-26T06:03:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 308 kB/s | 76 kB 00:00 2021-07-26T06:03:15.002 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 36 kB/s | 7.3 kB 00:00 2021-07-26T06:03:15.229 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 7.1 kB/s | 1.5 kB 00:00 2021-07-26T06:03:15.408 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - AppStream 52 MB/s | 8.2 MB 00:00 2021-07-26T06:03:16.795 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - Base 38 MB/s | 4.5 MB 00:00 2021-07-26T06:03:17.480 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - Extras 155 kB/s | 9.8 kB 00:00 2021-07-26T06:03:21.539 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Linux 8 - AppStream 2.1 MB/s | 8.2 MB 00:04 2021-07-26T06:03:24.207 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Linux 8 - BaseOS 3.3 MB/s | 4.5 MB 00:01 2021-07-26T06:03:25.172 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Linux 8 - Extras 30 kB/s | 9.8 kB 00:00 2021-07-26T06:03:25.348 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - PowerTools 19 MB/s | 2.3 MB 00:00 2021-07-26T06:03:26.690 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 11 MB/s | 10 MB 00:00 2021-07-26T06:03:28.829 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2021-07-26T06:03:30.852 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:03:30.877 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:03:30.877 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-07-26T06:03:30.877 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:03:30.878 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:03:30.878 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:17.0.0-6358.g197e564f.el8 ceph 7.6 M 2021-07-26T06:03:30.878 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2021-07-26T06:03:30.878 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:17.0.0-6358.g197e564f.el8 ceph 3.3 M 2021-07-26T06:03:30.879 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:17.0.0-6358.g197e564f.el8 ceph 3.7 M 2021-07-26T06:03:30.879 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-07-26T06:03:30.879 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:17.0.0-6358.g197e564f.el8 ceph 5.1 M 2021-07-26T06:03:30.879 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:17.0.0-6358.g197e564f.el8 ceph 19 M 2021-07-26T06:03:30.880 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:17.0.0-6358.g197e564f.el8 ceph 24 k 2021-07-26T06:03:30.880 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-07-26T06:03:30.880 INFO:teuthology.orchestra.run.smithi071.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-07-26T06:03:30.880 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:17.0.0-6358.g197e564f.el8 ceph 759 k 2021-07-26T06:03:30.880 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-07-26T06:03:30.881 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-07-26T06:03:30.881 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:17.0.0-6358.g197e564f.el8 ceph 463 k 2021-07-26T06:03:30.881 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-07-26T06:03:30.881 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:17.0.0-6358.g197e564f.el8 ceph 3.5 M 2021-07-26T06:03:30.882 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-07-26T06:03:30.882 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-07-26T06:03:30.882 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse 2021-07-26T06:03:30.882 INFO:teuthology.orchestra.run.smithi071.stdout: x86_64 2:17.0.0-6358.g197e564f.el8 ceph 45 k 2021-07-26T06:03:30.882 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:17.0.0-6358.g197e564f.el8 ceph 78 k 2021-07-26T06:03:30.883 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:17.0.0-6358.g197e564f.el8 ceph 178 k 2021-07-26T06:03:30.883 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-07-26T06:03:30.883 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:17.0.0-6358.g197e564f.el8 ceph 365 k 2021-07-26T06:03:30.883 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:17.0.0-6358.g197e564f.el8 ceph 346 k 2021-07-26T06:03:30.884 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:17.0.0-6358.g197e564f.el8 ceph 110 k 2021-07-26T06:03:30.884 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:03:30.884 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:03:30.884 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:03:30.885 INFO:teuthology.orchestra.run.smithi071.stdout:Install 21 Packages 2021-07-26T06:03:30.885 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2021-07-26T06:03:30.885 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:03:30.886 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 46 M 2021-07-26T06:03:30.886 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:03:31.577 INFO:teuthology.orchestra.run.smithi071.stdout:(1/23): ceph-base-17.0.0-6358.g197e564f.el8.x86 7.4 MB/s | 5.1 MB 00:00 2021-07-26T06:03:31.622 INFO:teuthology.orchestra.run.smithi071.stdout:(2/23): ceph-selinux-17.0.0-6358.g197e564f.el8. 548 kB/s | 24 kB 00:00 2021-07-26T06:03:31.726 INFO:teuthology.orchestra.run.smithi071.stdout:(3/23): libcephfs2-17.0.0-6358.g197e564f.el8.x8 7.1 MB/s | 759 kB 00:00 2021-07-26T06:03:31.781 INFO:teuthology.orchestra.run.smithi071.stdout:(4/23): ceph-radosgw-17.0.0-6358.g197e564f.el8. 8.4 MB/s | 7.6 MB 00:00 2021-07-26T06:03:31.784 INFO:teuthology.orchestra.run.smithi071.stdout:(5/23): libradosstriper1-17.0.0-6358.g197e564f. 7.7 MB/s | 463 kB 00:00 2021-07-26T06:03:31.829 INFO:teuthology.orchestra.run.smithi071.stdout:(6/23): python3-ceph-argparse-17.0.0-6358.g197e 1.0 MB/s | 45 kB 00:00 2021-07-26T06:03:31.878 INFO:teuthology.orchestra.run.smithi071.stdout:(7/23): python3-ceph-common-17.0.0-6358.g197e56 1.6 MB/s | 78 kB 00:00 2021-07-26T06:03:31.934 INFO:teuthology.orchestra.run.smithi071.stdout:(8/23): python3-cephfs-17.0.0-6358.g197e564f.el 3.2 MB/s | 178 kB 00:00 2021-07-26T06:03:31.998 INFO:teuthology.orchestra.run.smithi071.stdout:(9/23): python3-rados-17.0.0-6358.g197e564f.el8 5.5 MB/s | 365 kB 00:00 2021-07-26T06:03:32.065 INFO:teuthology.orchestra.run.smithi071.stdout:(10/23): python3-rbd-17.0.0-6358.g197e564f.el8. 5.1 MB/s | 346 kB 00:00 2021-07-26T06:03:32.092 INFO:teuthology.orchestra.run.smithi071.stdout:(11/23): librgw2-17.0.0-6358.g197e564f.el8.x86_ 11 MB/s | 3.5 MB 00:00 2021-07-26T06:03:32.101 INFO:teuthology.orchestra.run.smithi071.stdout:(12/23): python3-rgw-17.0.0-6358.g197e564f.el8. 3.0 MB/s | 110 kB 00:00 2021-07-26T06:03:32.127 INFO:teuthology.orchestra.run.smithi071.stdout:(13/23): librdkafka-0.11.4-1.el8.x86_64.rpm 11 MB/s | 353 kB 00:00 2021-07-26T06:03:32.127 INFO:teuthology.orchestra.run.smithi071.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 25 MB/s | 44 kB 00:00 2021-07-26T06:03:32.129 INFO:teuthology.orchestra.run.smithi071.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 30 MB/s | 47 kB 00:00 2021-07-26T06:03:32.132 INFO:teuthology.orchestra.run.smithi071.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 8.0 MB/s | 259 kB 00:00 2021-07-26T06:03:32.262 INFO:teuthology.orchestra.run.smithi071.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 2.3 MB/s | 306 kB 00:00 2021-07-26T06:03:32.265 INFO:teuthology.orchestra.run.smithi071.stdout:(18/23): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2021-07-26T06:03:32.284 INFO:teuthology.orchestra.run.smithi071.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 2.6 MB/s | 59 kB 00:00 2021-07-26T06:03:32.293 INFO:teuthology.orchestra.run.smithi071.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.7 MB/s | 75 kB 00:00 2021-07-26T06:03:32.483 INFO:teuthology.orchestra.run.smithi071.stdout:(21/23): ceph-common-17.0.0-6358.g197e564f.el8. 12 MB/s | 19 MB 00:01 2021-07-26T06:03:32.582 INFO:teuthology.orchestra.run.smithi071.stdout:(22/23): librados2-17.0.0-6358.g197e564f.el8.x8 11 MB/s | 3.3 MB 00:00 2021-07-26T06:03:32.609 INFO:teuthology.orchestra.run.smithi071.stdout:(23/23): librbd1-17.0.0-6358.g197e564f.el8.x86_ 12 MB/s | 3.7 MB 00:00 2021-07-26T06:03:32.613 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:03:32.613 INFO:teuthology.orchestra.run.smithi071.stdout:Total 26 MB/s | 46 MB 00:01 2021-07-26T06:03:32.616 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:03:32.660 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:03:32.660 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:03:32.956 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:03:32.959 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:03:34.008 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:03:34.242 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2021-07-26T06:03:34.428 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2021-07-26T06:03:34.800 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2021-07-26T06:03:35.011 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:17.0.0-6358.g197e564f.el8.x86_64 3/25 2021-07-26T06:03:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:17.0.0-6358.g197e564f.el8.x86_64 3/25 2021-07-26T06:03:35.324 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:17.0.0-6358.g197e564f.el8.x86_64 4/25 2021-07-26T06:03:35.519 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:17.0.0-6358.g197e564f.el8.x86_64 5/25 2021-07-26T06:03:35.921 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:17.0.0-6358.g197e564f.el8.x86_64 5/25 2021-07-26T06:03:36.094 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:17.0.0-6358.g197e564f.el8.x86_64 6/25 2021-07-26T06:03:36.158 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:17.0.0-6358.g197e564f.el8.x86_64 6/25 2021-07-26T06:03:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2021-07-26T06:03:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2021-07-26T06:03:36.911 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2021-07-26T06:03:37.085 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:17.0.0-6358.g197e564f.el8.x86_64 9/25 2021-07-26T06:03:37.164 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:17.0.0-6358.g197e564f.el8.x86_64 9/25 2021-07-26T06:03:37.292 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2021-07-26T06:03:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:17.0.0-6358.g197e564f.el8. 11/25 2021-07-26T06:03:37.639 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:17.0.0-6358.g197e564f.el8.x86_64 12/25 2021-07-26T06:03:37.836 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:17.0.0-6358.g197e564f.el8.x86_64 13/25 2021-07-26T06:03:38.009 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:17.0.0-6358.g197e564f.el8.x86_64 14/25 2021-07-26T06:03:38.194 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:17.0.0-6358.g197e564f.el8.x86_6 15/25 2021-07-26T06:03:38.251 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:17.0.0-6358.g197e564f.el8.x86_6 15/25 2021-07-26T06:03:38.436 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2021-07-26T06:03:38.627 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2021-07-26T06:03:38.771 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2021-07-26T06:03:38.919 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:17.0.0-6358.g197e564f.el8.x8 19/25 2021-07-26T06:03:43.010 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:17.0.0-6358.g197e564f.el8.x86_64 20/25 2021-07-26T06:03:43.210 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:17.0.0-6358.g197e564f.el8.x86_64 20/25 2021-07-26T06:03:43.784 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:17.0.0-6358.g197e564f.el8.x86_64 20/25 2021-07-26T06:03:43.999 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:17.0.0-6358.g197e564f.el8.x86_64 21/25 2021-07-26T06:03:44.129 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:17.0.0-6358.g197e564f.el8.x86_64 21/25 2021-07-26T06:03:44.234 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:17.0.0-6358.g197e564f.el8.x86_64 22/25 2021-07-26T06:03:59.482 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:17.0.0-6358.g197e564f.el8.x86_64 22/25 2021-07-26T06:03:59.482 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2021-07-26T06:03:59.482 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2021-07-26T06:03:59.483 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /dev 2021-07-26T06:03:59.483 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /run 2021-07-26T06:03:59.483 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2021-07-26T06:03:59.483 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2021-07-26T06:03:59.484 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2021-07-26T06:03:59.484 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2021-07-26T06:03:59.484 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /dev 2021-07-26T06:03:59.484 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:00.471 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:17.0.0-6358.g197e564f.el8.x86_64 23/25 2021-07-26T06:04:00.549 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-6358.g197e564f.el8.x86_64 23/25 2021-07-26T06:04:00.549 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-07-26T06:04:00.766 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-07-26T06:04:00.767 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2021-07-26T06:04:01.486 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2021-07-26T06:04:01.486 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:17.0.0-6358.g197e564f.el8.x86_64 1/25 2021-07-26T06:04:01.487 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:17.0.0-6358.g197e564f.el8.x86_64 2/25 2021-07-26T06:04:01.487 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:17.0.0-6358.g197e564f.el8.x86_64 3/25 2021-07-26T06:04:01.487 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:17.0.0-6358.g197e564f.el8.x86_64 4/25 2021-07-26T06:04:01.487 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:17.0.0-6358.g197e564f.el8.x86_64 5/25 2021-07-26T06:04:01.487 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:17.0.0-6358.g197e564f.el8.x86_6 6/25 2021-07-26T06:04:01.488 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:17.0.0-6358.g197e564f.el8.x86_64 7/25 2021-07-26T06:04:01.488 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:17.0.0-6358.g197e564f.el8. 8/25 2021-07-26T06:04:01.488 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:17.0.0-6358.g197e564f.el8.x8 9/25 2021-07-26T06:04:01.488 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:17.0.0-6358.g197e564f.el8.x86_64 10/25 2021-07-26T06:04:01.489 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:17.0.0-6358.g197e564f.el8.x86_64 11/25 2021-07-26T06:04:01.489 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:17.0.0-6358.g197e564f.el8.x86_64 12/25 2021-07-26T06:04:01.489 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:17.0.0-6358.g197e564f.el8.x86_64 13/25 2021-07-26T06:04:01.489 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2021-07-26T06:04:01.489 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2021-07-26T06:04:01.490 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2021-07-26T06:04:01.490 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2021-07-26T06:04:01.490 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2021-07-26T06:04:01.490 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2021-07-26T06:04:01.491 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2021-07-26T06:04:01.491 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2021-07-26T06:04:01.491 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:17.0.0-6358.g197e564f.el8.x86_64 22/25 2021-07-26T06:04:01.491 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2021-07-26T06:04:01.491 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:17.0.0-6358.g197e564f.el8.x86_64 24/25 2021-07-26T06:04:02.232 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2021-07-26T06:04:02.232 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:04:02.495 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:02.495 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2021-07-26T06:04:02.495 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.495 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.496 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:02.496 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:04:02.496 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.496 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.497 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.497 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.497 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-07-26T06:04:02.497 INFO:teuthology.orchestra.run.smithi071.stdout: leveldb-1.22-1.el8.x86_64 2021-07-26T06:04:02.497 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.498 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.2-3.el8.x86_64 2021-07-26T06:04:02.498 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-07-26T06:04:02.498 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.498 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-07-26T06:04:02.499 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.499 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.3.1-3.el8.x86_64 2021-07-26T06:04:02.499 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-07-26T06:04:02.499 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.499 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.500 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.500 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-07-26T06:04:02.500 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.500 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.500 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:02.501 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:02.501 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:04:02.714 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2021-07-26T06:04:03.250 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:35 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:04:04.152 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:04:04.153 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:04:04.153 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-07-26T06:04:04.154 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:04:04.154 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:04:04.154 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:17.0.0-6358.g197e564f.el8 ceph 44 M 2021-07-26T06:04:04.154 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-07-26T06:04:04.155 INFO:teuthology.orchestra.run.smithi071.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-07-26T06:04:04.155 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:17.0.0-6358.g197e564f.el8 ceph 166 k 2021-07-26T06:04:04.155 INFO:teuthology.orchestra.run.smithi071.stdout: libxslt x86_64 1.1.32-6.el8 CentOS-Base 250 k 2021-07-26T06:04:04.155 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-07-26T06:04:04.156 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-07-26T06:04:04.156 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-07-26T06:04:04.156 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:04.156 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:04:04.157 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:04:04.157 INFO:teuthology.orchestra.run.smithi071.stdout:Install 7 Packages 2021-07-26T06:04:04.157 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:04.158 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 45 M 2021-07-26T06:04:04.158 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 208 M 2021-07-26T06:04:04.158 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:04:04.196 INFO:teuthology.orchestra.run.smithi071.stdout:(1/7): jq-1.5-12.el8.x86_64.rpm 4.0 MB/s | 161 kB 00:00 2021-07-26T06:04:04.200 INFO:teuthology.orchestra.run.smithi071.stdout:(2/7): oniguruma-6.8.2-2.el8.x86_64.rpm 51 MB/s | 187 kB 00:00 2021-07-26T06:04:04.205 INFO:teuthology.orchestra.run.smithi071.stdout:(3/7): socat-1.7.3.3-2.el8.x86_64.rpm 62 MB/s | 302 kB 00:00 2021-07-26T06:04:04.208 INFO:teuthology.orchestra.run.smithi071.stdout:(4/7): libxslt-1.1.32-6.el8.x86_64.rpm 67 MB/s | 250 kB 00:00 2021-07-26T06:04:04.314 INFO:teuthology.orchestra.run.smithi071.stdout:(5/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 656 kB/s | 69 kB 00:00 2021-07-26T06:04:04.330 INFO:teuthology.orchestra.run.smithi071.stdout:(6/7): libcephsqlite-17.0.0-6358.g197e564f.el8. 953 kB/s | 166 kB 00:00 2021-07-26T06:04:06.722 INFO:teuthology.orchestra.run.smithi071.stdout:(7/7): ceph-test-17.0.0-6358.g197e564f.el8.x86_ 17 MB/s | 44 MB 00:02 2021-07-26T06:04:06.726 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:04:06.727 INFO:teuthology.orchestra.run.smithi071.stdout:Total 17 MB/s | 45 MB 00:02 2021-07-26T06:04:06.731 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:04:06.754 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:04:06.755 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:04:07.000 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:04:07.003 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:04:07.489 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:04:07.669 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libxslt-1.1.32-6.el8.x86_64 1/7 2021-07-26T06:04:07.896 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2021-07-26T06:04:08.057 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/7 2021-07-26T06:04:08.229 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-07-26T06:04:08.302 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/7 2021-07-26T06:04:08.437 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : jq-1.5-12.el8.x86_64 5/7 2021-07-26T06:04:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:17.0.0-6358.g197e564f.el8.x86_64 6/7 2021-07-26T06:04:12.907 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:17.0.0-6358.g197e564f.el8.x86_64 6/7 2021-07-26T06:04:13.126 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:17.0.0-6358.g197e564f.el8.x86_64 7/7 2021-07-26T06:04:13.390 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:17.0.0-6358.g197e564f.el8.x86_64 7/7 2021-07-26T06:04:13.390 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:17.0.0-6358.g197e564f.el8.x86_64 1/7 2021-07-26T06:04:13.390 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:17.0.0-6358.g197e564f.el8.x86_64 2/7 2021-07-26T06:04:13.391 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : jq-1.5-12.el8.x86_64 3/7 2021-07-26T06:04:13.391 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-07-26T06:04:13.391 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 5/7 2021-07-26T06:04:13.391 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libxslt-1.1.32-6.el8.x86_64 6/7 2021-07-26T06:04:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 7/7 2021-07-26T06:04:13.743 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:04:14.109 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:14.110 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:04:14.110 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:17.0.0-6358.g197e564f.el8.x86_64 jq-1.5-12.el8.x86_64 2021-07-26T06:04:14.110 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:17.0.0-6358.g197e564f.el8.x86_64 libxslt-1.1.32-6.el8.x86_64 2021-07-26T06:04:14.111 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.3.3-2.el8.x86_64 2021-07-26T06:04:14.111 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-07-26T06:04:14.111 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:14.111 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:04:14.274 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2021-07-26T06:04:14.800 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:46 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:04:16.291 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:04:16.295 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2021-07-26T06:04:16.295 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-07-26T06:04:16.295 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2021-07-26T06:04:16.295 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:04:16.296 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:17.0.0-6358.g197e564f.el8 ceph 6.4 k 2021-07-26T06:04:16.296 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-07-26T06:04:16.296 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:17.0.0-6358.g197e564f.el8 ceph 2.2 M 2021-07-26T06:04:16.296 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:17.0.0-6358.g197e564f.el8 ceph 1.5 M 2021-07-26T06:04:16.297 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:17.0.0-6358.g197e564f.el8 ceph-noarch 211 k 2021-07-26T06:04:16.297 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:17.0.0-6358.g197e564f.el8 ceph 3.8 M 2021-07-26T06:04:16.297 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:17.0.0-6358.g197e564f.el8 ceph 17 M 2021-07-26T06:04:16.297 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-07-26T06:04:16.298 INFO:teuthology.orchestra.run.smithi071.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-07-26T06:04:16.298 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-07-26T06:04:16.298 INFO:teuthology.orchestra.run.smithi071.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-07-26T06:04:16.298 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-07-26T06:04:16.299 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-07-26T06:04:16.299 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 3.2.1-4.el8 CentOS-Base 559 k 2021-07-26T06:04:16.299 INFO:teuthology.orchestra.run.smithi071.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-07-26T06:04:16.299 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-07-26T06:04:16.299 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-07-26T06:04:16.300 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-07-26T06:04:16.300 INFO:teuthology.orchestra.run.smithi071.stdout: python3-lxml x86_64 4.2.3-2.el8 CentOS-AppStream 1.5 M 2021-07-26T06:04:16.300 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-07-26T06:04:16.300 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-07-26T06:04:16.300 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-07-26T06:04:16.301 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-07-26T06:04:16.301 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-07-26T06:04:16.301 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-07-26T06:04:16.301 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-07-26T06:04:16.302 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-07-26T06:04:16.302 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-07-26T06:04:16.302 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-07-26T06:04:16.302 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-07-26T06:04:16.302 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-07-26T06:04:16.303 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-07-26T06:04:16.303 INFO:teuthology.orchestra.run.smithi071.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-07-26T06:04:16.303 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-07-26T06:04:16.303 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-07-26T06:04:16.304 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-07-26T06:04:16.304 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-07-26T06:04:16.304 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-07-26T06:04:16.304 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-07-26T06:04:16.304 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-07-26T06:04:16.305 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2021-07-26T06:04:16.305 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-07-26T06:04:16.305 INFO:teuthology.orchestra.run.smithi071.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-07-26T06:04:16.305 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:16.305 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:04:16.306 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2021-07-26T06:04:16.306 INFO:teuthology.orchestra.run.smithi071.stdout:Install 41 Packages 2021-07-26T06:04:16.306 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:16.307 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 31 M 2021-07-26T06:04:16.307 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 116 M 2021-07-26T06:04:16.307 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:04:16.479 INFO:teuthology.orchestra.run.smithi071.stdout:(1/41): ceph-17.0.0-6358.g197e564f.el8.x86_64.r 36 kB/s | 6.4 kB 00:00 2021-07-26T06:04:17.286 INFO:teuthology.orchestra.run.smithi071.stdout:(2/41): ceph-mgr-17.0.0-6358.g197e564f.el8.x86_ 1.5 MB/s | 1.5 MB 00:00 2021-07-26T06:04:17.814 INFO:teuthology.orchestra.run.smithi071.stdout:(3/41): ceph-mds-17.0.0-6358.g197e564f.el8.x86_ 1.5 MB/s | 2.2 MB 00:01 2021-07-26T06:04:17.884 INFO:teuthology.orchestra.run.smithi071.stdout:(4/41): ceph-mgr-modules-core-17.0.0-6358.g197e 3.0 MB/s | 211 kB 00:00 2021-07-26T06:04:17.918 INFO:teuthology.orchestra.run.smithi071.stdout:(5/41): python3-html5lib-0.999999999-6.el8.noar 6.0 MB/s | 214 kB 00:00 2021-07-26T06:04:17.937 INFO:teuthology.orchestra.run.smithi071.stdout:(6/41): python3-lxml-4.2.3-2.el8.x86_64.rpm 83 MB/s | 1.5 MB 00:00 2021-07-26T06:04:17.940 INFO:teuthology.orchestra.run.smithi071.stdout:(7/41): python3-mako-1.0.6-13.el8.noarch.rpm 57 MB/s | 157 kB 00:00 2021-07-26T06:04:17.941 INFO:teuthology.orchestra.run.smithi071.stdout:(8/41): python3-markupsafe-0.23-19.el8.x86_64.r 30 MB/s | 39 kB 00:00 2021-07-26T06:04:17.943 INFO:teuthology.orchestra.run.smithi071.stdout:(9/41): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 51 MB/s | 103 kB 00:00 2021-07-26T06:04:17.945 INFO:teuthology.orchestra.run.smithi071.stdout:(10/41): python3-pytz-2017.2-9.el8.noarch.rpm 36 MB/s | 54 kB 00:00 2021-07-26T06:04:17.946 INFO:teuthology.orchestra.run.smithi071.stdout:(11/41): python3-webencodings-0.5.1-6.el8.noarc 21 MB/s | 27 kB 00:00 2021-07-26T06:04:17.953 INFO:teuthology.orchestra.run.smithi071.stdout:(12/41): python3-werkzeug-0.12.2-4.el8.noarch.r 75 MB/s | 457 kB 00:00 2021-07-26T06:04:17.957 INFO:teuthology.orchestra.run.smithi071.stdout:(13/41): python3-cffi-1.11.5-5.el8.x86_64.rpm 55 MB/s | 237 kB 00:00 2021-07-26T06:04:17.960 INFO:teuthology.orchestra.run.smithi071.stdout:(14/41): python3-chardet-3.0.4-7.el8.noarch.rpm 62 MB/s | 195 kB 00:00 2021-07-26T06:04:17.967 INFO:teuthology.orchestra.run.smithi071.stdout:(15/41): python3-cryptography-3.2.1-4.el8.x86_6 77 MB/s | 559 kB 00:00 2021-07-26T06:04:17.970 INFO:teuthology.orchestra.run.smithi071.stdout:(16/41): python3-idna-2.5-5.el8.noarch.rpm 47 MB/s | 97 kB 00:00 2021-07-26T06:04:17.972 INFO:teuthology.orchestra.run.smithi071.stdout:(17/41): python3-pycparser-2.14-14.el8.noarch.r 50 MB/s | 109 kB 00:00 2021-07-26T06:04:17.973 INFO:teuthology.orchestra.run.smithi071.stdout:(18/41): python3-pysocks-1.6.8-3.el8.noarch.rpm 28 MB/s | 34 kB 00:00 2021-07-26T06:04:17.976 INFO:teuthology.orchestra.run.smithi071.stdout:(19/41): python3-requests-2.20.0-2.1.el8_1.noar 53 MB/s | 123 kB 00:00 2021-07-26T06:04:17.979 INFO:teuthology.orchestra.run.smithi071.stdout:(20/41): python3-urllib3-1.24.2-5.el8.noarch.rp 59 MB/s | 177 kB 00:00 2021-07-26T06:04:18.129 INFO:teuthology.orchestra.run.smithi071.stdout:(21/41): ceph-mon-17.0.0-6358.g197e564f.el8.x86 2.3 MB/s | 3.8 MB 00:01 2021-07-26T06:04:18.234 INFO:teuthology.orchestra.run.smithi071.stdout:(22/41): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 171 kB/s | 44 kB 00:00 2021-07-26T06:04:18.298 INFO:teuthology.orchestra.run.smithi071.stdout:(23/41): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.6 MB/s | 173 kB 00:00 2021-07-26T06:04:18.337 INFO:teuthology.orchestra.run.smithi071.stdout:(24/41): python3-beautifulsoup4-4.6.3-2.el8.1.n 890 kB/s | 185 kB 00:00 2021-07-26T06:04:18.362 INFO:teuthology.orchestra.run.smithi071.stdout:(25/41): python3-cherrypy-18.4.0-1.el8.noarch.r 5.8 MB/s | 384 kB 00:00 2021-07-26T06:04:18.365 INFO:teuthology.orchestra.run.smithi071.stdout:(26/41): python3-cssselect-0.9.2-10.el8.noarch. 1.4 MB/s | 40 kB 00:00 2021-07-26T06:04:18.384 INFO:teuthology.orchestra.run.smithi071.stdout:(27/41): python3-jaraco-6.2-6.el8.noarch.rpm 514 kB/s | 11 kB 00:00 2021-07-26T06:04:18.392 INFO:teuthology.orchestra.run.smithi071.stdout:(28/41): python3-jaraco-functools-2.0-4.el8.noa 648 kB/s | 18 kB 00:00 2021-07-26T06:04:18.406 INFO:teuthology.orchestra.run.smithi071.stdout:(29/41): python3-logutils-0.3.5-11.el8.noarch.r 2.2 MB/s | 49 kB 00:00 2021-07-26T06:04:18.419 INFO:teuthology.orchestra.run.smithi071.stdout:(30/41): python3-more-itertools-7.2.0-3.el8.noa 2.1 MB/s | 59 kB 00:00 2021-07-26T06:04:18.447 INFO:teuthology.orchestra.run.smithi071.stdout:(31/41): python3-portend-2.6-1.el8.noarch.rpm 578 kB/s | 16 kB 00:00 2021-07-26T06:04:18.450 INFO:teuthology.orchestra.run.smithi071.stdout:(32/41): python3-pecan-1.3.2-9.el8.noarch.rpm 6.4 MB/s | 283 kB 00:00 2021-07-26T06:04:18.471 INFO:teuthology.orchestra.run.smithi071.stdout:(33/41): python3-singledispatch-3.4.0.3-18.el8. 1.1 MB/s | 24 kB 00:00 2021-07-26T06:04:18.473 INFO:teuthology.orchestra.run.smithi071.stdout:(34/41): python3-simplegeneric-0.8.1-17.el8.noa 697 kB/s | 19 kB 00:00 2021-07-26T06:04:18.493 INFO:teuthology.orchestra.run.smithi071.stdout:(35/41): python3-tempora-1.14.1-5.el8.noarch.rp 1.3 MB/s | 29 kB 00:00 2021-07-26T06:04:18.501 INFO:teuthology.orchestra.run.smithi071.stdout:(36/41): python3-trustme-0.6.0-4.el8.noarch.rpm 1.0 MB/s | 27 kB 00:00 2021-07-26T06:04:18.537 INFO:teuthology.orchestra.run.smithi071.stdout:(37/41): python3-waitress-1.2.1-2.el8.1.noarch. 5.5 MB/s | 241 kB 00:00 2021-07-26T06:04:18.555 INFO:teuthology.orchestra.run.smithi071.stdout:(38/41): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.5 MB/s | 251 kB 00:00 2021-07-26T06:04:18.559 INFO:teuthology.orchestra.run.smithi071.stdout:(39/41): python3-webtest-2.0.33-1.el8.noarch.rp 3.8 MB/s | 85 kB 00:00 2021-07-26T06:04:18.582 INFO:teuthology.orchestra.run.smithi071.stdout:(40/41): python3-zc-lockfile-2.0-2.el8.noarch.r 854 kB/s | 23 kB 00:00 2021-07-26T06:04:21.371 INFO:teuthology.orchestra.run.smithi071.stdout:(41/41): ceph-osd-17.0.0-6358.g197e564f.el8.x86 4.1 MB/s | 17 MB 00:04 2021-07-26T06:04:21.376 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:04:21.376 INFO:teuthology.orchestra.run.smithi071.stdout:Total 6.0 MB/s | 31 MB 00:05 2021-07-26T06:04:21.391 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:04:21.435 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:04:21.436 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:04:21.733 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:04:21.736 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:04:23.194 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:04:23.400 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/41 2021-07-26T06:04:23.586 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/41 2021-07-26T06:04:23.759 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-idna-2.5-5.el8.noarch 3/41 2021-07-26T06:04:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/41 2021-07-26T06:04:24.191 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/41 2021-07-26T06:04:24.354 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/41 2021-07-26T06:04:24.553 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/41 2021-07-26T06:04:24.712 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/41 2021-07-26T06:04:24.882 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/41 2021-07-26T06:04:25.055 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/41 2021-07-26T06:04:25.192 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/41 2021-07-26T06:04:25.445 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/41 2021-07-26T06:04:25.695 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 13/41 2021-07-26T06:04:25.911 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/41 2021-07-26T06:04:26.306 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/41 2021-07-26T06:04:26.554 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 16/41 2021-07-26T06:04:26.733 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 17/41 2021-07-26T06:04:27.004 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 18/41 2021-07-26T06:04:27.226 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 19/41 2021-07-26T06:04:27.511 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 20/41 2021-07-26T06:04:27.765 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 21/41 2021-07-26T06:04:28.038 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 22/41 2021-07-26T06:04:28.202 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/41 2021-07-26T06:04:28.420 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 24/41 2021-07-26T06:04:28.645 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 25/41 2021-07-26T06:04:28.978 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 26/41 2021-07-26T06:04:29.215 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-lxml-4.2.3-2.el8.x86_64 27/41 2021-07-26T06:04:29.430 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 28/41 2021-07-26T06:04:29.636 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 29/41 2021-07-26T06:04:29.821 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 30/41 2021-07-26T06:04:30.097 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-2.6-1.el8.noarch 31/41 2021-07-26T06:04:30.353 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 32/41 2021-07-26T06:04:30.596 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/41 2021-07-26T06:04:30.971 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 34/41 2021-07-26T06:04:31.324 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 35/41 2021-07-26T06:04:31.681 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-6358.g197e564f.el8. 36/41 2021-07-26T06:04:31.872 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:17.0.0-6358.g197e564f.el8.x86_64 37/41 2021-07-26T06:04:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:17.0.0-6358.g197e564f.el8.x86_64 37/41 2021-07-26T06:04:33.747 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:17.0.0-6358.g197e564f.el8.x86_64 38/41 2021-07-26T06:04:34.130 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:17.0.0-6358.g197e564f.el8.x86_64 38/41 2021-07-26T06:04:34.305 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:17.0.0-6358.g197e564f.el8.x86_64 39/41 2021-07-26T06:04:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:17.0.0-6358.g197e564f.el8.x86_64 39/41 2021-07-26T06:04:34.723 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:17.0.0-6358.g197e564f.el8.x86_64 40/41 2021-07-26T06:04:34.740 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:17.0.0-6358.g197e564f.el8.x86_64 40/41 2021-07-26T06:04:34.905 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:17.0.0-6358.g197e564f.el8.x86_64 41/41 2021-07-26T06:04:35.267 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:17.0.0-6358.g197e564f.el8.x86_64 41/41 2021-07-26T06:04:35.268 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:17.0.0-6358.g197e564f.el8.x86_64 1/41 2021-07-26T06:04:35.268 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:17.0.0-6358.g197e564f.el8.x86_64 2/41 2021-07-26T06:04:35.268 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:17.0.0-6358.g197e564f.el8.x86_64 3/41 2021-07-26T06:04:35.268 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:17.0.0-6358.g197e564f.el8.x86_64 4/41 2021-07-26T06:04:35.269 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:17.0.0-6358.g197e564f.el8.x86_64 5/41 2021-07-26T06:04:35.269 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-6358.g197e564f.el8. 6/41 2021-07-26T06:04:35.269 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/41 2021-07-26T06:04:35.269 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-lxml-4.2.3-2.el8.x86_64 8/41 2021-07-26T06:04:35.269 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/41 2021-07-26T06:04:35.270 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/41 2021-07-26T06:04:35.270 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 11/41 2021-07-26T06:04:35.270 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/41 2021-07-26T06:04:35.270 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/41 2021-07-26T06:04:35.271 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/41 2021-07-26T06:04:35.271 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 15/41 2021-07-26T06:04:35.271 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 16/41 2021-07-26T06:04:35.271 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 17/41 2021-07-26T06:04:35.272 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-idna-2.5-5.el8.noarch 18/41 2021-07-26T06:04:35.272 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 19/41 2021-07-26T06:04:35.272 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 20/41 2021-07-26T06:04:35.272 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 21/41 2021-07-26T06:04:35.272 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 22/41 2021-07-26T06:04:35.273 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 23/41 2021-07-26T06:04:35.273 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/41 2021-07-26T06:04:35.273 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 25/41 2021-07-26T06:04:35.273 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 26/41 2021-07-26T06:04:35.273 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 27/41 2021-07-26T06:04:35.275 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 28/41 2021-07-26T06:04:35.275 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 29/41 2021-07-26T06:04:35.276 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 30/41 2021-07-26T06:04:35.276 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 31/41 2021-07-26T06:04:35.276 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 32/41 2021-07-26T06:04:35.276 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-2.6-1.el8.noarch 33/41 2021-07-26T06:04:35.276 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 34/41 2021-07-26T06:04:35.277 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 35/41 2021-07-26T06:04:35.277 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 36/41 2021-07-26T06:04:35.277 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 37/41 2021-07-26T06:04:35.277 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 38/41 2021-07-26T06:04:35.278 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 39/41 2021-07-26T06:04:35.278 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 40/41 2021-07-26T06:04:35.983 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 41/41 2021-07-26T06:04:35.984 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:04:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:36.238 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:04:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:17.0.0-6358.g197e564f.el8.noarch 2021-07-26T06:04:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-07-26T06:04:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-07-26T06:04:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-07-26T06:04:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-07-26T06:04:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-07-26T06:04:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-07-26T06:04:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2021-07-26T06:04:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-07-26T06:04:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-07-26T06:04:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: python3-idna-2.5-5.el8.noarch 2021-07-26T06:04:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-6.2-6.el8.noarch 2021-07-26T06:04:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-07-26T06:04:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-07-26T06:04:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: python3-lxml-4.2.3-2.el8.x86_64 2021-07-26T06:04:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.0.6-13.el8.noarch 2021-07-26T06:04:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-07-26T06:04:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-07-26T06:04:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-07-26T06:04:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-2.6-1.el8.noarch 2021-07-26T06:04:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-07-26T06:04:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.14-14.el8.noarch 2021-07-26T06:04:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-07-26T06:04:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2017.2-9.el8.noarch 2021-07-26T06:04:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-07-26T06:04:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-07-26T06:04:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-07-26T06:04:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-07-26T06:04:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-07-26T06:04:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-07-26T06:04:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-07-26T06:04:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-07-26T06:04:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-07-26T06:04:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-07-26T06:04:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-07-26T06:04:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-07-26T06:04:36.248 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:36.248 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:04:36.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2021-07-26T06:04:36.994 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:08 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:04:37.652 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:17.0.0-6358.g197e564f.el8.x86_64 is already installed. 2021-07-26T06:04:37.887 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:04:37.888 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-07-26T06:04:37.888 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:04:37.960 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2021-07-26T06:04:38.497 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:10 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:04:39.408 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:04:39.409 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:04:39.409 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-07-26T06:04:39.409 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:04:39.410 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:04:39.410 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:17.0.0-6358.g197e564f.el8 ceph-noarch 68 k 2021-07-26T06:04:39.410 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:39.410 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:04:39.410 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:04:39.411 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-07-26T06:04:39.411 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:39.411 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 68 k 2021-07-26T06:04:39.411 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 299 k 2021-07-26T06:04:39.411 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:04:39.699 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-17.0.0-6358.g197e564f.el8.noarch.rpm 234 kB/s | 68 kB 00:00 2021-07-26T06:04:39.700 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:04:39.700 INFO:teuthology.orchestra.run.smithi071.stdout:Total 233 kB/s | 68 kB 00:00 2021-07-26T06:04:39.700 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:04:39.706 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:04:39.707 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:04:39.708 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:04:39.711 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:04:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:04:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:17.0.0-6358.g197e564f.el8.noarch 1/1 2021-07-26T06:04:42.110 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:17.0.0-6358.g197e564f.el8.noarch 1/1 2021-07-26T06:04:42.285 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:17.0.0-6358.g197e564f.el8.noarch 1/1 2021-07-26T06:04:42.554 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:17.0.0-6358.g197e564f.el8.noarch 1/1 2021-07-26T06:04:42.554 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:04:42.799 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:42.800 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:04:42.800 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:17.0.0-6358.g197e564f.el8.noarch 2021-07-26T06:04:42.800 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:42.800 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:04:43.402 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2021-07-26T06:04:44.009 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:16 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:04:45.312 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:04:45.313 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:04:45.314 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2021-07-26T06:04:45.314 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:04:45.314 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:04:45.314 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-6358.g197e564f.el8 ceph 166 k 2021-07-26T06:04:45.315 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:45.315 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:04:45.315 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:04:45.315 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-07-26T06:04:45.316 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:45.316 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 166 k 2021-07-26T06:04:45.316 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 519 k 2021-07-26T06:04:45.316 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:04:45.536 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-17.0.0-6358.g197e56 749 kB/s | 166 kB 00:00 2021-07-26T06:04:45.537 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:04:45.537 INFO:teuthology.orchestra.run.smithi071.stdout:Total 743 kB/s | 166 kB 00:00 2021-07-26T06:04:45.538 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:04:45.549 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:04:45.550 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:04:45.598 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:04:45.601 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:04:45.710 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:04:45.938 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-6358.g197e564f. 1/1 2021-07-26T06:04:46.306 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-6358.g197e564f. 1/1 2021-07-26T06:04:46.530 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-6358.g197e564f. 1/1 2021-07-26T06:04:46.530 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:04:46.756 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:46.757 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:04:46.757 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:04:46.757 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:46.757 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:04:46.929 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2021-07-26T06:04:47.476 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:19 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:04:48.154 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:17.0.0-6358.g197e564f.el8.x86_64 is already installed. 2021-07-26T06:04:48.380 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:04:48.381 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-07-26T06:04:48.382 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:04:48.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-dashboard 2021-07-26T06:04:48.976 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:20 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:04:49.869 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:04:49.869 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:04:49.870 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-07-26T06:04:49.870 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:04:49.870 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:04:49.870 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:17.0.0-6358.g197e564f.el8 ceph-noarch 1.5 M 2021-07-26T06:04:49.871 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-07-26T06:04:49.871 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:17.0.0-6358.g197e564f.el8 ceph-noarch 23 k 2021-07-26T06:04:49.871 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:17.0.0-6358.g197e564f.el8 ceph-noarch 8.9 k 2021-07-26T06:04:49.871 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-07-26T06:04:49.871 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-07-26T06:04:49.872 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-07-26T06:04:49.872 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:49.872 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:04:49.872 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:04:49.872 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2021-07-26T06:04:49.873 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:49.873 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 1.8 M 2021-07-26T06:04:49.874 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 7.4 M 2021-07-26T06:04:49.874 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:04:49.983 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): ceph-prometheus-alerts-17.0.0-6358.g197e 80 kB/s | 8.9 kB 00:00 2021-07-26T06:04:49.996 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): ceph-grafana-dashboards-17.0.0-6358.g197 188 kB/s | 23 kB 00:00 2021-07-26T06:04:50.030 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 913 kB/s | 43 kB 00:00 2021-07-26T06:04:50.064 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 498 kB/s | 34 kB 00:00 2021-07-26T06:04:50.185 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2021-07-26T06:04:50.505 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): ceph-mgr-dashboard-17.0.0-6358.g197e564f 2.4 MB/s | 1.5 MB 00:00 2021-07-26T06:04:50.508 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:04:50.508 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.9 MB/s | 1.8 MB 00:00 2021-07-26T06:04:50.509 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:04:50.517 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:04:50.517 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:04:50.574 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:04:50.578 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:04:50.820 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:04:51.035 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-07-26T06:04:51.255 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-07-26T06:04:51.497 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-07-26T06:04:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-6358.g197e564f.el8.n 4/6 2021-07-26T06:04:51.982 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-6358.g197e564f.el8. 5/6 2021-07-26T06:04:52.184 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-6358.g197e564f.el8.noarc 6/6 2021-07-26T06:04:52.415 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-6358.g197e564f.el8.noarc 6/6 2021-07-26T06:04:52.416 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-6358.g197e564f.el8. 1/6 2021-07-26T06:04:52.416 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-6358.g197e564f.el8.noarc 2/6 2021-07-26T06:04:52.416 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-6358.g197e564f.el8.n 3/6 2021-07-26T06:04:52.416 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-07-26T06:04:52.416 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-07-26T06:04:52.702 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-07-26T06:04:52.702 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:04:52.947 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:52.947 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:04:52.947 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:17.0.0-6358.g197e564f.el8.noarch 2021-07-26T06:04:52.948 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:17.0.0-6358.g197e564f.el8.noarch 2021-07-26T06:04:52.948 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:17.0.0-6358.g197e564f.el8.noarch 2021-07-26T06:04:52.948 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-07-26T06:04:52.948 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-07-26T06:04:52.948 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.4.1-12.el8.noarch 2021-07-26T06:04:52.949 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:52.949 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:04:53.017 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-diskprediction-local 2021-07-26T06:04:53.555 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:25 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:04:54.455 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:04:54.456 INFO:teuthology.orchestra.run.smithi071.stdout:===================================================================================================== 2021-07-26T06:04:54.456 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-07-26T06:04:54.457 INFO:teuthology.orchestra.run.smithi071.stdout:===================================================================================================== 2021-07-26T06:04:54.457 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:04:54.457 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-6358.g197e564f.el8 ceph-noarch 7.4 M 2021-07-26T06:04:54.457 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-07-26T06:04:54.457 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 8.4.1-1.el8 CentOS-Base 639 k 2021-07-26T06:04:54.458 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 8.4.1-1.el8 CentOS-Base 169 k 2021-07-26T06:04:54.458 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.12-1.el8 CentOS-AppStream 4.6 M 2021-07-26T06:04:54.458 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-threads x86_64 0.3.12-1.el8 CentOS-AppStream 4.7 M 2021-07-26T06:04:54.458 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-07-26T06:04:54.459 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-07-26T06:04:54.459 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.0.0-20.module_el8.4.0+790+083e3d81 CentOS-AppStream 15 M 2021-07-26T06:04:54.459 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:54.459 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:04:54.459 INFO:teuthology.orchestra.run.smithi071.stdout:===================================================================================================== 2021-07-26T06:04:54.460 INFO:teuthology.orchestra.run.smithi071.stdout:Install 8 Packages 2021-07-26T06:04:54.460 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:04:54.461 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 36 M 2021-07-26T06:04:54.461 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 197 M 2021-07-26T06:04:54.461 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:04:54.555 INFO:teuthology.orchestra.run.smithi071.stdout:(1/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 49 MB/s | 4.7 MB 00:00 2021-07-26T06:04:54.607 INFO:teuthology.orchestra.run.smithi071.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 72 MB/s | 3.7 MB 00:00 2021-07-26T06:04:54.612 INFO:teuthology.orchestra.run.smithi071.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 47 MB/s | 225 kB 00:00 2021-07-26T06:04:54.784 INFO:teuthology.orchestra.run.smithi071.stdout:(4/8): openblas-0.3.12-1.el8.x86_64.rpm 14 MB/s | 4.6 MB 00:00 2021-07-26T06:04:54.890 INFO:teuthology.orchestra.run.smithi071.stdout:(5/8): python3-scipy-1.0.0-20.module_el8.4.0+79 52 MB/s | 15 MB 00:00 2021-07-26T06:04:54.894 INFO:teuthology.orchestra.run.smithi071.stdout:(6/8): libgfortran-8.4.1-1.el8.x86_64.rpm 5.6 MB/s | 639 kB 00:00 2021-07-26T06:04:54.895 INFO:teuthology.orchestra.run.smithi071.stdout:(7/8): libquadmath-8.4.1-1.el8.x86_64.rpm 36 MB/s | 169 kB 00:00 2021-07-26T06:04:56.485 INFO:teuthology.orchestra.run.smithi071.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-635 3.7 MB/s | 7.4 MB 00:02 2021-07-26T06:04:56.489 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:04:56.490 INFO:teuthology.orchestra.run.smithi071.stdout:Total 18 MB/s | 36 MB 00:02 2021-07-26T06:04:56.499 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:04:56.519 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:04:56.519 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:04:56.907 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:04:56.910 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:04:57.453 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:04:57.608 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2021-07-26T06:04:57.784 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2021-07-26T06:04:57.918 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2021-07-26T06:04:58.443 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2021-07-26T06:04:58.609 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.12-1.el8.x86_64 3/8 2021-07-26T06:04:59.153 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 3/8 2021-07-26T06:04:59.318 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-07-26T06:05:00.041 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-07-26T06:05:00.359 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-07-26T06:05:00.842 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-07-26T06:05:02.860 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-07-26T06:05:04.090 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x 7/8 2021-07-26T06:05:04.258 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-6358.g197e564 8/8 2021-07-26T06:05:04.524 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-6358.g197e564 8/8 2021-07-26T06:05:04.524 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-6358.g197e564 1/8 2021-07-26T06:05:04.524 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 2/8 2021-07-26T06:05:04.525 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-07-26T06:05:04.525 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-07-26T06:05:04.525 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-07-26T06:05:04.525 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x 6/8 2021-07-26T06:05:04.525 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 7/8 2021-07-26T06:05:04.796 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 8/8 2021-07-26T06:05:04.797 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:05:05.010 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:05.011 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:05:05.011 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:17.0.0-6358.g197e564f.el8.noarch 2021-07-26T06:05:05.011 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-8.4.1-1.el8.x86_64 2021-07-26T06:05:05.011 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-8.4.1-1.el8.x86_64 2021-07-26T06:05:05.012 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.12-1.el8.x86_64 2021-07-26T06:05:05.012 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-07-26T06:05:05.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-07-26T06:05:05.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-07-26T06:05:05.013 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x86_64 2021-07-26T06:05:05.013 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:05.013 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:05:05.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-rook 2021-07-26T06:05:05.641 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:37 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:05:06.969 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:05:06.970 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2021-07-26T06:05:06.971 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-07-26T06:05:06.971 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2021-07-26T06:05:06.971 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:05:06.972 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:17.0.0-6358.g197e564f.el8 ceph-noarch 41 k 2021-07-26T06:05:06.972 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-07-26T06:05:06.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-07-26T06:05:06.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-07-26T06:05:06.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-07-26T06:05:06.973 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-07-26T06:05:06.973 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-07-26T06:05:06.973 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-07-26T06:05:06.973 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-07-26T06:05:06.974 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-07-26T06:05:06.974 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-07-26T06:05:06.974 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-07-26T06:05:06.975 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-07-26T06:05:06.975 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-07-26T06:05:06.975 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:06.975 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:05:06.976 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2021-07-26T06:05:06.976 INFO:teuthology.orchestra.run.smithi071.stdout:Install 13 Packages 2021-07-26T06:05:06.976 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:06.977 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 1.9 M 2021-07-26T06:05:06.977 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 25 M 2021-07-26T06:05:06.977 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:05:07.121 INFO:teuthology.orchestra.run.smithi071.stdout:(1/13): ceph-mgr-rook-17.0.0-6358.g197e564f.el8 277 kB/s | 41 kB 00:00 2021-07-26T06:05:35.683 INFO:teuthology.orchestra.run.smithi071.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 946 B/s | 27 kB 00:28 2021-07-26T06:05:35.684 INFO:teuthology.orchestra.run.smithi071.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.4 kB/s | 126 kB 00:28 2021-07-26T06:05:35.685 INFO:teuthology.orchestra.run.smithi071.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 38 MB/s | 110 kB 00:00 2021-07-26T06:05:35.691 INFO:teuthology.orchestra.run.smithi071.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 7.3 MB/s | 43 kB 00:00 2021-07-26T06:05:35.695 INFO:teuthology.orchestra.run.smithi071.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 15 MB/s | 155 kB 00:00 2021-07-26T06:05:35.697 INFO:teuthology.orchestra.run.smithi071.stdout:(7/13): python3-jsonpointer-1.10-11.el8.noarch. 710 B/s | 20 kB 00:28 2021-07-26T06:05:35.757 INFO:teuthology.orchestra.run.smithi071.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 506 kB/s | 33 kB 00:00 2021-07-26T06:05:35.861 INFO:teuthology.orchestra.run.smithi071.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 99 kB/s | 16 kB 00:00 2021-07-26T06:05:35.893 INFO:teuthology.orchestra.run.smithi071.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 416 kB/s | 81 kB 00:00 2021-07-26T06:05:35.918 INFO:teuthology.orchestra.run.smithi071.stdout:(11/13): python3-rsa-3.4.2-16.el8.noarch.rpm 1.3 MB/s | 73 kB 00:00 2021-07-26T06:05:35.921 INFO:teuthology.orchestra.run.smithi071.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 2.1 MB/s | 61 kB 00:00 2021-07-26T06:05:35.929 INFO:teuthology.orchestra.run.smithi071.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.8 MB/s | 1.2 MB 00:00 2021-07-26T06:05:35.933 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:05:35.934 INFO:teuthology.orchestra.run.smithi071.stdout:Total 68 kB/s | 1.9 MB 00:28 2021-07-26T06:05:35.939 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:05:35.955 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:05:35.955 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:05:36.033 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:05:36.036 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:05:36.541 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:05:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-07-26T06:05:36.983 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-07-26T06:05:37.229 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-07-26T06:05:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-07-26T06:05:37.633 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-07-26T06:05:37.858 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-07-26T06:05:38.132 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-07-26T06:05:38.357 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-07-26T06:05:39.418 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-07-26T06:05:39.641 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-07-26T06:05:39.858 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-07-26T06:05:40.051 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-07-26T06:05:40.258 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:17.0.0-6358.g197e564f.el8.noarch 13/13 2021-07-26T06:05:40.502 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-6358.g197e564f.el8.noarch 13/13 2021-07-26T06:05:40.502 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:17.0.0-6358.g197e564f.el8.noarch 1/13 2021-07-26T06:05:40.503 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-07-26T06:05:40.503 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-07-26T06:05:40.503 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-07-26T06:05:40.503 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-07-26T06:05:40.504 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-07-26T06:05:40.504 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-07-26T06:05:40.504 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-07-26T06:05:40.504 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-07-26T06:05:40.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-07-26T06:05:40.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-07-26T06:05:40.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-07-26T06:05:40.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-07-26T06:05:40.846 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:05:41.071 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:41.072 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:05:41.072 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:17.0.0-6358.g197e564f.el8.noarch 2021-07-26T06:05:41.072 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-07-26T06:05:41.072 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-07-26T06:05:41.073 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-07-26T06:05:41.073 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-07-26T06:05:41.073 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-07-26T06:05:41.073 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-07-26T06:05:41.073 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-07-26T06:05:41.074 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-07-26T06:05:41.074 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-07-26T06:05:41.074 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-07-26T06:05:41.074 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-07-26T06:05:41.075 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-07-26T06:05:41.075 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:41.075 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:05:41.158 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-cephadm 2021-07-26T06:05:41.704 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:13 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:05:42.596 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:05:42.597 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:05:42.597 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-07-26T06:05:42.597 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:05:42.598 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:05:42.598 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:17.0.0-6358.g197e564f.el8 ceph-noarch 90 k 2021-07-26T06:05:42.598 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-07-26T06:05:42.598 INFO:teuthology.orchestra.run.smithi071.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-07-26T06:05:42.598 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-07-26T06:05:42.599 INFO:teuthology.orchestra.run.smithi071.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-07-26T06:05:42.599 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-07-26T06:05:42.599 INFO:teuthology.orchestra.run.smithi071.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-07-26T06:05:42.599 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:42.600 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:05:42.600 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:05:42.600 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2021-07-26T06:05:42.600 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:42.601 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 5.8 M 2021-07-26T06:05:42.601 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 24 M 2021-07-26T06:05:42.602 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:05:42.658 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 9.0 MB/s | 538 kB 00:00 2021-07-26T06:05:42.695 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): python3-babel-2.5.1-5.el8.noarch.rpm 50 MB/s | 4.8 MB 00:00 2021-07-26T06:05:42.797 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 153 kB/s | 21 kB 00:00 2021-07-26T06:05:42.816 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): ceph-mgr-cephadm-17.0.0-6358.g197e564f.e 415 kB/s | 90 kB 00:00 2021-07-26T06:05:42.850 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): python3-remoto-1.1.4-4.el8.noarch.rpm 893 kB/s | 48 kB 00:00 2021-07-26T06:05:42.928 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.5 MB/s | 355 kB 00:00 2021-07-26T06:05:42.930 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:05:42.931 INFO:teuthology.orchestra.run.smithi071.stdout:Total 17 MB/s | 5.8 MB 00:00 2021-07-26T06:05:42.933 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:05:42.941 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:05:42.942 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:05:43.022 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:05:43.025 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:05:43.284 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:05:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-07-26T06:05:43.813 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-07-26T06:05:44.713 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-07-26T06:05:45.084 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2021-07-26T06:05:45.373 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-07-26T06:05:45.607 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-6358.g197e564f.el8.noarch 6/6 2021-07-26T06:05:45.880 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-6358.g197e564f.el8.noarch 6/6 2021-07-26T06:05:45.880 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-6358.g197e564f.el8.noarch 1/6 2021-07-26T06:05:45.880 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2021-07-26T06:05:45.881 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2021-07-26T06:05:45.881 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-07-26T06:05:45.881 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-07-26T06:05:46.148 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-07-26T06:05:46.148 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:05:46.376 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:46.376 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:05:46.376 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:17.0.0-6358.g197e564f.el8.noarch 2021-07-26T06:05:46.377 INFO:teuthology.orchestra.run.smithi071.stdout: python3-apipkg-1.5-6.el8.noarch 2021-07-26T06:05:46.377 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.5.1-5.el8.noarch 2021-07-26T06:05:46.377 INFO:teuthology.orchestra.run.smithi071.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-07-26T06:05:46.377 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-07-26T06:05:46.378 INFO:teuthology.orchestra.run.smithi071.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-07-26T06:05:46.378 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:46.378 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:05:46.448 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2021-07-26T06:05:46.997 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:18 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:05:47.895 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:05:47.896 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:05:47.897 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-07-26T06:05:47.897 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:05:47.897 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:05:47.897 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:17.0.0-6358.g197e564f.el8 ceph 843 k 2021-07-26T06:05:47.898 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-07-26T06:05:47.898 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-07-26T06:05:47.898 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:47.898 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:05:47.899 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:05:47.899 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2021-07-26T06:05:47.899 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:47.899 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 926 k 2021-07-26T06:05:47.899 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 2.8 M 2021-07-26T06:05:47.900 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:05:47.933 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.3 MB/s | 82 kB 00:00 2021-07-26T06:05:48.162 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-fuse-17.0.0-6358.g197e564f.el8.x86_ 3.1 MB/s | 843 kB 00:00 2021-07-26T06:05:48.163 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:05:48.163 INFO:teuthology.orchestra.run.smithi071.stdout:Total 3.4 MB/s | 926 kB 00:00 2021-07-26T06:05:48.164 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:05:48.185 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:05:48.185 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:05:48.262 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:05:48.265 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:05:48.423 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:05:48.729 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-07-26T06:05:48.948 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:17.0.0-6358.g197e564f.el8.x86_64 2/2 2021-07-26T06:05:49.295 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:17.0.0-6358.g197e564f.el8.x86_64 2/2 2021-07-26T06:05:49.296 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:17.0.0-6358.g197e564f.el8.x86_64 1/2 2021-07-26T06:05:49.554 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-07-26T06:05:49.554 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:05:49.811 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:49.811 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:05:49.811 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:17.0.0-6358.g197e564f.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-07-26T06:05:49.811 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:49.812 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:05:49.966 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2021-07-26T06:05:50.530 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:22 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:05:51.416 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:05:51.417 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:05:51.417 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2021-07-26T06:05:51.418 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:05:51.418 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:05:51.418 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:17.0.0-6358.g197e564f.el8 ceph 125 k 2021-07-26T06:05:51.418 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:51.418 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:05:51.419 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:05:51.419 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-07-26T06:05:51.419 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:51.419 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 125 k 2021-07-26T06:05:51.420 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 432 k 2021-07-26T06:05:51.420 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:05:51.587 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-17.0.0-6358.g197e564f.el8.x86_64 737 kB/s | 125 kB 00:00 2021-07-26T06:05:51.588 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:05:51.588 INFO:teuthology.orchestra.run.smithi071.stdout:Total 731 kB/s | 125 kB 00:00 2021-07-26T06:05:51.588 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:05:51.594 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:05:51.594 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:05:51.646 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:05:51.649 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:05:51.762 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:05:51.972 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:05:52.237 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:05:52.487 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:05:52.487 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:05:52.718 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:52.718 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:05:52.718 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:05:52.719 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:52.719 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:05:52.904 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2021-07-26T06:05:53.459 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:25 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:05:54.128 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:17.0.0-6358.g197e564f.el8.x86_64 is already installed. 2021-07-26T06:05:54.357 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:05:54.358 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-07-26T06:05:54.359 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:05:54.414 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2021-07-26T06:05:54.963 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:26 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:05:55.879 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:05:55.880 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:05:55.881 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2021-07-26T06:05:55.881 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:05:55.881 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:05:55.881 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:17.0.0-6358.g197e564f.el8 ceph 24 k 2021-07-26T06:05:55.882 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:55.882 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:05:55.882 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:05:55.882 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-07-26T06:05:55.883 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:55.883 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 24 k 2021-07-26T06:05:55.883 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 102 k 2021-07-26T06:05:55.883 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:05:56.220 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-17.0.0-6358.g197e564f.el8.x86_6 71 kB/s | 24 kB 00:00 2021-07-26T06:05:56.221 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:05:56.221 INFO:teuthology.orchestra.run.smithi071.stdout:Total 70 kB/s | 24 kB 00:00 2021-07-26T06:05:56.221 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:05:56.224 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:05:56.225 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:05:56.226 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:05:56.229 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:05:56.277 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:05:56.487 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:05:56.708 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:05:56.927 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:05:56.928 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:05:57.150 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:57.150 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:05:57.151 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:05:57.151 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:05:57.151 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:05:57.258 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2021-07-26T06:05:57.806 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:29 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:05:58.460 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:17.0.0-6358.g197e564f.el8.x86_64 is already installed. 2021-07-26T06:05:58.696 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:05:58.697 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-07-26T06:05:58.698 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:05:58.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2021-07-26T06:05:59.307 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:31 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:06:00.001 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:17.0.0-6358.g197e564f.el8.x86_64 is already installed. 2021-07-26T06:06:00.233 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:06:00.234 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-07-26T06:06:00.234 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:06:00.370 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2021-07-26T06:06:00.922 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:32 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:06:01.609 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:17.0.0-6358.g197e564f.el8.x86_64 is already installed. 2021-07-26T06:06:01.839 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:06:01.840 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-07-26T06:06:01.840 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:06:01.898 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2021-07-26T06:06:02.448 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:34 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:06:03.131 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:17.0.0-6358.g197e564f.el8.x86_64 is already installed. 2021-07-26T06:06:03.362 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:06:03.363 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-07-26T06:06:03.363 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:06:03.421 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2021-07-26T06:06:03.968 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:35 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:06:04.625 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:17.0.0-6358.g197e564f.el8.x86_64 is already installed. 2021-07-26T06:06:04.861 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:06:04.862 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-07-26T06:06:04.862 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:06:04.932 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2021-07-26T06:06:05.481 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:37 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:06:06.133 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:17.0.0-6358.g197e564f.el8.x86_64 is already installed. 2021-07-26T06:06:06.369 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:06:06.370 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-07-26T06:06:06.370 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:06:06.434 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2021-07-26T06:06:06.983 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:38 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:06:07.889 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:06:07.890 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:06:07.890 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2021-07-26T06:06:07.890 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:06:07.890 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:06:07.891 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:17.0.0-6358.g197e564f.el8 ceph 91 k 2021-07-26T06:06:07.891 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:06:07.891 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:06:07.891 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:06:07.891 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-07-26T06:06:07.892 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:06:07.892 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 91 k 2021-07-26T06:06:07.892 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 264 k 2021-07-26T06:06:07.892 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:06:08.037 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-17.0.0-6358.g197e564f.el8.x86_64.rpm 622 kB/s | 91 kB 00:00 2021-07-26T06:06:08.038 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:06:08.038 INFO:teuthology.orchestra.run.smithi071.stdout:Total 617 kB/s | 91 kB 00:00 2021-07-26T06:06:08.038 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:06:08.049 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:06:08.049 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:06:08.100 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:06:08.103 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:06:08.212 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:06:08.418 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:06:08.656 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:06:08.910 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:06:08.911 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:06:09.155 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:06:09.155 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:06:09.156 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:06:09.156 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:06:09.156 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:06:09.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2021-07-26T06:06:09.870 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:41 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:06:10.899 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:06:10.900 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:06:10.900 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2021-07-26T06:06:10.900 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:06:10.900 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:06:10.901 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:17.0.0-6358.g197e564f.el8 ceph 3.7 M 2021-07-26T06:06:10.901 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:06:10.901 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:06:10.901 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:06:10.901 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-07-26T06:06:10.902 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:06:10.902 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.7 M 2021-07-26T06:06:10.902 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 17 M 2021-07-26T06:06:10.902 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:06:11.403 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-17.0.0-6358.g197e564f.el8.x86_64.rpm 7.3 MB/s | 3.7 MB 00:00 2021-07-26T06:06:11.403 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:06:11.404 INFO:teuthology.orchestra.run.smithi071.stdout:Total 7.3 MB/s | 3.7 MB 00:00 2021-07-26T06:06:11.404 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:06:11.417 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:06:11.417 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:06:11.479 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:06:11.483 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:06:11.939 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:06:12.161 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:06:12.519 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:06:12.762 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:06:12.762 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:06:13.049 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:06:13.049 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:06:13.049 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:06:13.050 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:06:13.050 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:06:13.226 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2021-07-26T06:06:13.785 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:45 ago on Mon 26 Jul 2021 06:03:28 AM UTC. 2021-07-26T06:06:14.690 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-07-26T06:06:14.691 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:06:14.692 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2021-07-26T06:06:14.692 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:06:14.692 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-07-26T06:06:14.693 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:17.0.0-6358.g197e564f.el8 ceph 172 k 2021-07-26T06:06:14.693 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:06:14.693 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-07-26T06:06:14.693 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-07-26T06:06:14.694 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-07-26T06:06:14.694 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:06:14.694 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 172 k 2021-07-26T06:06:14.695 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 539 k 2021-07-26T06:06:14.695 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-07-26T06:06:14.863 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-17.0.0-6358.g197e564f.el8.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2021-07-26T06:06:14.863 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-07-26T06:06:14.864 INFO:teuthology.orchestra.run.smithi071.stdout:Total 994 kB/s | 172 kB 00:00 2021-07-26T06:06:14.864 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-07-26T06:06:14.876 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-07-26T06:06:14.876 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-07-26T06:06:14.927 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-07-26T06:06:14.930 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-07-26T06:06:15.069 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-07-26T06:06:15.248 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:06:15.493 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:06:15.744 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:17.0.0-6358.g197e564f.el8.x86_64 1/1 2021-07-26T06:06:15.744 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2021-07-26T06:06:15.982 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:06:15.982 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-07-26T06:06:15.982 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:17.0.0-6358.g197e564f.el8.x86_64 2021-07-26T06:06:15.983 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-07-26T06:06:15.983 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-07-26T06:06:16.233 DEBUG:teuthology.parallel:result is None 2021-07-26T06:06:16.234 DEBUG:teuthology.orchestra.run.smithi071:> sudo lsb_release -is 2021-07-26T06:06:16.272 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS 2021-07-26T06:06:16.273 DEBUG:teuthology.misc:System to be installed: CentOS 2021-07-26T06:06:16.274 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=197e564fdaee12596de1dc8f3253d1704ae66d5d 2021-07-26T06:06:16.438 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-07-26T06:06:16.457 INFO:teuthology.orchestra.run.smithi071.stdout:17.0.0-6358.g197e564f.el8 2021-07-26T06:06:16.458 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-6358.g197e564f.el8 2021-07-26T06:06:16.458 INFO:teuthology.task.install:The correct ceph version 17.0.0-6358.g197e564f is installed. 2021-07-26T06:06:16.459 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-07-26T06:06:16.459 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-07-26T06:06:16.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-07-26T06:06:16.520 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-07-26T06:06:16.521 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-07-26T06:06:16.521 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2021-07-26T06:06:16.584 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-07-26T06:06:16.648 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-07-26T06:06:16.649 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-07-26T06:06:16.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2021-07-26T06:06:16.711 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-07-26T06:06:16.774 INFO:teuthology.run_tasks:Running task workunit... 2021-07-26T06:06:16.791 INFO:tasks.workunit:Pulling workunits from ref 197e564fdaee12596de1dc8f3253d1704ae66d5d 2021-07-26T06:06:16.792 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-07-26T06:06:16.793 INFO:tasks.workunit:timeout=3h 2021-07-26T06:06:16.793 INFO:tasks.workunit:cleanup=True 2021-07-26T06:06:16.793 DEBUG:teuthology.orchestra.run.smithi071:> stat -- /home/ubuntu/cephtest/mnt.0 2021-07-26T06:06:16.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-26T06:06:16.828 INFO:teuthology.orchestra.run.smithi071.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2021-07-26T06:06:16.829 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -- /home/ubuntu/cephtest/mnt.0 2021-07-26T06:06:16.883 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2021-07-26T06:06:16.883 DEBUG:teuthology.orchestra.run.smithi071:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2021-07-26T06:06:16.937 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 197e564fdaee12596de1dc8f3253d1704ae66d5d 2021-07-26T06:06:16.991 INFO:tasks.workunit.client.0.smithi071.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-07-26T06:06:57.502 INFO:tasks.workunit.client.0.smithi071.stderr:Note: switching to '197e564fdaee12596de1dc8f3253d1704ae66d5d'. 2021-07-26T06:06:57.503 INFO:tasks.workunit.client.0.smithi071.stderr: 2021-07-26T06:06:57.503 INFO:tasks.workunit.client.0.smithi071.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2021-07-26T06:06:57.503 INFO:tasks.workunit.client.0.smithi071.stderr:changes and commit them, and you can discard any commits you make in this 2021-07-26T06:06:57.504 INFO:tasks.workunit.client.0.smithi071.stderr:state without impacting any branches by switching back to a branch. 2021-07-26T06:06:57.504 INFO:tasks.workunit.client.0.smithi071.stderr: 2021-07-26T06:06:57.504 INFO:tasks.workunit.client.0.smithi071.stderr:If you want to create a new branch to retain commits you create, you may 2021-07-26T06:06:57.504 INFO:tasks.workunit.client.0.smithi071.stderr:do so (now or later) by using -c with the switch command. Example: 2021-07-26T06:06:57.505 INFO:tasks.workunit.client.0.smithi071.stderr: 2021-07-26T06:06:57.505 INFO:tasks.workunit.client.0.smithi071.stderr: git switch -c 2021-07-26T06:06:57.505 INFO:tasks.workunit.client.0.smithi071.stderr: 2021-07-26T06:06:57.505 INFO:tasks.workunit.client.0.smithi071.stderr:Or undo this operation with: 2021-07-26T06:06:57.505 INFO:tasks.workunit.client.0.smithi071.stderr: 2021-07-26T06:06:57.506 INFO:tasks.workunit.client.0.smithi071.stderr: git switch - 2021-07-26T06:06:57.506 INFO:tasks.workunit.client.0.smithi071.stderr: 2021-07-26T06:06:57.506 INFO:tasks.workunit.client.0.smithi071.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2021-07-26T06:06:57.506 INFO:tasks.workunit.client.0.smithi071.stderr: 2021-07-26T06:06:57.507 INFO:tasks.workunit.client.0.smithi071.stderr:HEAD is now at 197e564fdae Merge branch 'ups-eliminate-rollfwd' of https://github.com/LiumxNL/ceph into wip-kefu-testing-2021-07-26-1205 2021-07-26T06:06:57.508 DEBUG:teuthology.orchestra.run.smithi071:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2021-07-26T06:06:57.563 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-07-26T06:06:57.564 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2021-07-26T06:06:57.617 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2021-07-26T06:06:57.618 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2021-07-26T06:06:57.618 DEBUG:teuthology.orchestra.run.smithi071:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2021-07-26T06:06:57.681 INFO:tasks.workunit.client.0.smithi071.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:06:57.689 INFO:tasks.workunit.client.0.smithi071.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:06:57.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-07-26T06:06:57.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-07-26T06:06:57.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:06:57.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:06:57.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:06:57.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:06:57.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:06:57.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:06:57.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-recovery-scrub 2021-07-26T06:06:57.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2021-07-26T06:06:57.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2021-07-26T06:06:57.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2021-07-26T06:06:57.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2021-07-26T06:06:57.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2021-07-26T06:06:57.692 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2021-07-26T06:06:57.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none ' 2021-07-26T06:06:57.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2021-07-26T06:06:57.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:28: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:06:57.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: set 2021-07-26T06:06:57.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:06:57.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: local 'funcs=TEST_recovery_scrub_1 2021-07-26T06:06:57.699 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_recovery_scrub_2' 2021-07-26T06:06:57.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: for func in $funcs 2021-07-26T06:06:57.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2021-07-26T06:06:57.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:38: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2021-07-26T06:06:57.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local poolname=test 2021-07-26T06:06:57.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:41: TEST_recovery_scrub_1: TESTDATA=testdata.27826 2021-07-26T06:06:57.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: OSDS=4 2021-07-26T06:06:57.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: PGS=1 2021-07-26T06:06:57.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: OBJECTS=100 2021-07-26T06:06:57.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: ERRORS=0 2021-07-26T06:06:57.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2021-07-26T06:06:57.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2021-07-26T06:06:57.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2021-07-26T06:06:57.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2021-07-26T06:06:57.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:06:57.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2021-07-26T06:06:57.703 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:06:57.703 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:06:57.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:06:57.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:06:57.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:06:57.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:06:57.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:06:57.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:06:57.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:06:57.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:06:57.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:06:57.709 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:06:57.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:06:57.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:06:57.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:06:57.710 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:06:57.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:06:57.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:06:57.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2021-07-26T06:06:57.711 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:06:57.711 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:06:57.712 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:06:57.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27826 2021-07-26T06:06:57.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:06:57.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:06:57.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2021-07-26T06:06:57.713 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:06:57.713 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:06:57.713 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:06:57.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27826 2021-07-26T06:06:57.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:06:57.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:06:57.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:06:57.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:06:57.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2021-07-26T06:06:57.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2021-07-26T06:06:57.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2021-07-26T06:06:57.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:06:57.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:06:57.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:06:57.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2021-07-26T06:06:57.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2021-07-26T06:06:58.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:06:58.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:06:58.171 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:06:58.171 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:06:58.171 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:06:58.171 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:06:58.172 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:06:58.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2021-07-26T06:06:58.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:06:58.390 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:06:58.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:06:58.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:06:58.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:06:58.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:06:58.392 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:06:58.392 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:06:58.392 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:06:58.392 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:06:58.393 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:06:58.393 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:06:58.393 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27826/ceph-mon.a.asok 2021-07-26T06:06:58.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:06:58.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.27826/ceph-mon.a.asok config get fsid 2021-07-26T06:06:58.488 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:06:58.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:06:58.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:06:58.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:06:58.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:06:58.490 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:06:58.490 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:06:58.490 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:06:58.490 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:06:58.490 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:06:58.491 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:06:58.491 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27826/ceph-mon.a.asok 2021-07-26T06:06:58.491 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:06:58.491 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.27826/ceph-mon.a.asok config get mon_host 2021-07-26T06:06:58.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2021-07-26T06:06:58.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2021-07-26T06:06:58.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:06:58.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:06:58.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:06:58.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2021-07-26T06:06:58.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:06:59.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:06:59.400 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:06:59.400 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:06:59.400 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:06:59.401 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:06:59.401 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:06:59.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:06:59.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:06:59.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:06:59.422 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: expr 4 - 1 2021-07-26T06:06:59.423 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: seq 0 3 2021-07-26T06:06:59.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:06:59.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2021-07-26T06:06:59.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2021-07-26T06:06:59.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:06:59.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:06:59.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:06:59.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2021-07-26T06:06:59.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2021-07-26T06:06:59.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:06:59.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:06:59.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:06:59.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2021-07-26T06:06:59.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2021-07-26T06:06:59.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:06:59.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:06:59.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:06:59.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2021-07-26T06:06:59.440 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:06:59.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:06:59.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:06:59.441 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:06:59.441 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:06:59.441 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:06:59.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:06:59.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:06:59.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:06:59.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:06:59.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:06:59.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2021-07-26T06:06:59.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2021-07-26T06:06:59.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:06:59.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:06:59.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:06:59.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:06:59.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2021-07-26T06:06:59.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2021-07-26T06:06:59.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:06:59.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d5a08579-528b-4ed1-b576-df1693645f24 2021-07-26T06:06:59.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d5a08579-528b-4ed1-b576-df1693645f24' 2021-07-26T06:06:59.445 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 d5a08579-528b-4ed1-b576-df1693645f24 2021-07-26T06:06:59.446 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:06:59.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADUf5gTjvGGhAAgl0aXnZG0yQdCZkY/t1lYA== 2021-07-26T06:06:59.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADUf5gTjvGGhAAgl0aXnZG0yQdCZkY/t1lYA=="}' 2021-07-26T06:06:59.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d5a08579-528b-4ed1-b576-df1693645f24 -i td/osd-recovery-scrub/0/new.json 2021-07-26T06:06:59.756 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:06:59.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2021-07-26T06:06:59.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --mkfs --key AQADUf5gTjvGGhAAgl0aXnZG0yQdCZkY/t1lYA== --osd-uuid d5a08579-528b-4ed1-b576-df1693645f24 2021-07-26T06:06:59.790 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:06:59.790+0000 7ff9fffe80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:06:59.792 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:06:59.792+0000 7ff9fffe80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:06:59.793 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:06:59.793+0000 7ff9fffe80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:06:59.793 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:06:59.793+0000 7ff9fffe80c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2021-07-26T06:06:59.793 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:06:59.793+0000 7ff9fffe80c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2021-07-26T06:06:59.794 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:06:59.793+0000 7ff9fffe80c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2021-07-26T06:06:59.891 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:06:59.891+0000 7ff9fffe80c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2021-07-26T06:07:05.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2021-07-26T06:07:05.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:07:05.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:07:05.074 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:07:05.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:07:05.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:07:05.404 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:07:05.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false 2021-07-26T06:07:05.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:07:05.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:07:05.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:07:05.424 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:05.424+0000 7f994b9980c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:05.425 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:05.425+0000 7f994b9980c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:05.426 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:05.426+0000 7f994b9980c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:05.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:07:05.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:07:05.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:07:05.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:07:05.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:07:05.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:05.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:07:05.671 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:07:05.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:05.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:07:05.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:05.953 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:05.953+0000 7f994b9980c0 -1 Falling back to public interface 2021-07-26T06:07:06.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:06.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:06.933 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:07:06.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:07:06.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:06.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:07:07.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:08.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:08.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:08.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:07:08.232 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:07:08.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:08.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:07:08.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:08.588 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:08.589+0000 7f994b9980c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:07:09.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:09.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:09.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:07:09.507 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:07:09.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:09.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:07:09.851 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/28948,v1:127.0.0.1:6803/28948] [v2:127.0.0.1:6804/28948,v1:127.0.0.1:6805/28948] exists,up d5a08579-528b-4ed1-b576-df1693645f24 2021-07-26T06:07:09.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:07:09.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:07:09.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:07:09.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:07:09.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2021-07-26T06:07:09.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2021-07-26T06:07:09.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:07:09.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:07:09.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:07:09.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2021-07-26T06:07:09.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2021-07-26T06:07:09.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:07:09.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:07:09.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:07:09.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2021-07-26T06:07:09.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2021-07-26T06:07:09.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:07:09.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:07:09.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:07:09.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2021-07-26T06:07:09.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:07:09.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:07:09.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:07:09.861 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:07:09.861 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:07:09.861 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:07:09.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:07:09.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:07:09.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:07:09.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:07:09.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:07:09.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2021-07-26T06:07:09.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2021-07-26T06:07:09.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:07:09.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:07:09.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:07:09.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:07:09.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2021-07-26T06:07:09.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2021-07-26T06:07:09.865 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:07:09.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6d3fedcf-9263-4a2c-8267-ec994af48076 2021-07-26T06:07:09.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6d3fedcf-9263-4a2c-8267-ec994af48076' 2021-07-26T06:07:09.865 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 6d3fedcf-9263-4a2c-8267-ec994af48076 2021-07-26T06:07:09.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:07:09.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANUf5g/o7BMxAAQj6ePOY35M90D+OE0/v9Lg== 2021-07-26T06:07:09.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANUf5g/o7BMxAAQj6ePOY35M90D+OE0/v9Lg=="}' 2021-07-26T06:07:09.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6d3fedcf-9263-4a2c-8267-ec994af48076 -i td/osd-recovery-scrub/1/new.json 2021-07-26T06:07:10.230 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:07:10.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2021-07-26T06:07:10.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --mkfs --key AQANUf5g/o7BMxAAQj6ePOY35M90D+OE0/v9Lg== --osd-uuid 6d3fedcf-9263-4a2c-8267-ec994af48076 2021-07-26T06:07:10.264 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:10.265+0000 7f8884f8e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:10.266 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:10.267+0000 7f8884f8e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:10.266 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:10.267+0000 7f8884f8e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:10.267 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:10.268+0000 7f8884f8e0c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2021-07-26T06:07:10.267 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:10.268+0000 7f8884f8e0c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2021-07-26T06:07:10.268 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:10.268+0000 7f8884f8e0c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2021-07-26T06:07:10.322 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:10.323+0000 7f8884f8e0c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2021-07-26T06:07:15.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2021-07-26T06:07:15.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:07:15.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:07:15.774 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:07:15.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:07:16.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:07:16.113 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:07:16.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:07:16.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false 2021-07-26T06:07:16.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:07:16.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:07:16.133 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:16.134+0000 7f11892ca0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:16.135 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:16.136+0000 7f11892ca0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:16.136 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:16.137+0000 7f11892ca0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:16.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:07:16.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:07:16.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:07:16.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:07:16.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:07:16.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:16.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:07:16.414 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:07:16.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:16.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:07:16.654 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:16.654+0000 7f11892ca0c0 -1 Falling back to public interface 2021-07-26T06:07:16.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:17.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:17.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:17.699 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:07:17.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:07:17.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:17.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:07:17.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:18.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:18.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:18.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:07:18.975 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:07:18.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:18.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:07:19.125 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:19.126+0000 7f11892ca0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:07:19.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:20.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:20.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:20.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:07:20.284 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:07:20.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:20.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:07:20.556 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/30347,v1:127.0.0.1:6811/30347] [v2:127.0.0.1:6812/30347,v1:127.0.0.1:6813/30347] exists,up 6d3fedcf-9263-4a2c-8267-ec994af48076 2021-07-26T06:07:20.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:07:20.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:07:20.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:07:20.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:07:20.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2021-07-26T06:07:20.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2021-07-26T06:07:20.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:07:20.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:07:20.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:07:20.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2021-07-26T06:07:20.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2021-07-26T06:07:20.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:07:20.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:07:20.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:07:20.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2021-07-26T06:07:20.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2021-07-26T06:07:20.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:07:20.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:07:20.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:07:20.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2021-07-26T06:07:20.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:07:20.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:07:20.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:07:20.566 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:07:20.566 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:07:20.567 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:07:20.567 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:07:20.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:07:20.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:07:20.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:07:20.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:07:20.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2021-07-26T06:07:20.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2021-07-26T06:07:20.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:07:20.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:07:20.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:07:20.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:07:20.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2021-07-26T06:07:20.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2021-07-26T06:07:20.570 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:07:20.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d7cf3926-49d4-41eb-805a-8cd87b449811 2021-07-26T06:07:20.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d7cf3926-49d4-41eb-805a-8cd87b449811' 2021-07-26T06:07:20.571 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 d7cf3926-49d4-41eb-805a-8cd87b449811 2021-07-26T06:07:20.571 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:07:20.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAYUf5gKHBCIhAApocUStIk6o537/hyUH03sg== 2021-07-26T06:07:20.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAYUf5gKHBCIhAApocUStIk6o537/hyUH03sg=="}' 2021-07-26T06:07:20.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d7cf3926-49d4-41eb-805a-8cd87b449811 -i td/osd-recovery-scrub/2/new.json 2021-07-26T06:07:20.961 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:07:20.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2021-07-26T06:07:20.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --mkfs --key AQAYUf5gKHBCIhAApocUStIk6o537/hyUH03sg== --osd-uuid d7cf3926-49d4-41eb-805a-8cd87b449811 2021-07-26T06:07:20.995 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:20.996+0000 7f778acee0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:21.053 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:21.054+0000 7f778acee0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:21.054 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:21.055+0000 7f778acee0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:21.055 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:21.055+0000 7f778acee0c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2021-07-26T06:07:21.055 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:21.055+0000 7f778acee0c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2021-07-26T06:07:21.055 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:21.055+0000 7f778acee0c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2021-07-26T06:07:21.111 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:21.112+0000 7f778acee0c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2021-07-26T06:07:26.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2021-07-26T06:07:26.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:07:26.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:07:26.835 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:07:26.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:07:27.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:07:27.176 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:07:27.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false 2021-07-26T06:07:27.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:07:27.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:07:27.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:07:27.196 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:27.197+0000 7f9669afb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:27.197 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:27.198+0000 7f9669afb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:27.198 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:27.199+0000 7f9669afb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:27.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:07:27.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:07:27.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:07:27.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:07:27.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:07:27.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:27.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:07:27.518 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:07:27.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:27.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:07:27.717 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:27.717+0000 7f9669afb0c0 -1 Falling back to public interface 2021-07-26T06:07:27.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:28.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:28.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:28.784 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:07:28.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:07:28.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:28.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:07:29.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:30.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:30.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:30.307 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:07:30.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:07:30.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:30.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:07:30.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:31.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:31.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:31.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:07:31.650 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:07:31.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:31.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:07:32.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:32.214 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:32.215+0000 7f9669afb0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:07:33.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:33.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:33.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:07:33.195 INFO:tasks.workunit.client.0.smithi071.stdout:4 2021-07-26T06:07:33.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:33.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:07:33.516 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/31754,v1:127.0.0.1:6819/31754] [v2:127.0.0.1:6820/31754,v1:127.0.0.1:6821/31754] exists,up d7cf3926-49d4-41eb-805a-8cd87b449811 2021-07-26T06:07:33.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:07:33.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:07:33.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:07:33.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:07:33.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2021-07-26T06:07:33.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2021-07-26T06:07:33.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:07:33.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T06:07:33.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:07:33.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2021-07-26T06:07:33.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2021-07-26T06:07:33.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:07:33.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:07:33.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:07:33.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2021-07-26T06:07:33.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2021-07-26T06:07:33.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:07:33.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:07:33.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:07:33.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2021-07-26T06:07:33.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:07:33.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:07:33.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:07:33.528 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:07:33.529 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:07:33.529 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:07:33.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:07:33.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:07:33.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:07:33.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:07:33.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:07:33.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2021-07-26T06:07:33.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2021-07-26T06:07:33.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:07:33.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:07:33.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:07:33.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:07:33.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2021-07-26T06:07:33.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2021-07-26T06:07:33.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:07:33.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4ed67cf2-6ba5-46b4-a188-c7f5346e051c 2021-07-26T06:07:33.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 4ed67cf2-6ba5-46b4-a188-c7f5346e051c' 2021-07-26T06:07:33.533 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 4ed67cf2-6ba5-46b4-a188-c7f5346e051c 2021-07-26T06:07:33.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:07:33.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlUf5gJu/yHxAA4ki7Rwz/ZadsuOnCnqwOVQ== 2021-07-26T06:07:33.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlUf5gJu/yHxAA4ki7Rwz/ZadsuOnCnqwOVQ=="}' 2021-07-26T06:07:33.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4ed67cf2-6ba5-46b4-a188-c7f5346e051c -i td/osd-recovery-scrub/3/new.json 2021-07-26T06:07:33.915 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:07:33.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2021-07-26T06:07:33.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --mkfs --key AQAlUf5gJu/yHxAA4ki7Rwz/ZadsuOnCnqwOVQ== --osd-uuid 4ed67cf2-6ba5-46b4-a188-c7f5346e051c 2021-07-26T06:07:33.949 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:33.950+0000 7fddcc5920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:33.956 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:33.957+0000 7fddcc5920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:33.957 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:33.958+0000 7fddcc5920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:33.958 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:33.959+0000 7fddcc5920c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2021-07-26T06:07:33.958 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:33.959+0000 7fddcc5920c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2021-07-26T06:07:33.959 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:33.959+0000 7fddcc5920c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2021-07-26T06:07:34.034 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:34.035+0000 7fddcc5920c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2021-07-26T06:07:39.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2021-07-26T06:07:39.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:07:39.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T06:07:39.652 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd3 key to auth repository 2021-07-26T06:07:39.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:07:39.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T06:07:39.987 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.3 2021-07-26T06:07:39.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false 2021-07-26T06:07:39.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:07:39.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:07:39.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:07:40.008 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:40.009+0000 7f0b17f970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:40.010 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:40.010+0000 7f0b17f970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:40.010 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:40.011+0000 7f0b17f970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:40.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T06:07:40.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:07:40.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:07:40.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:07:40.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:07:40.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:40.287 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:07:40.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:07:40.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:40.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:07:40.527 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:40.528+0000 7f0b17f970c0 -1 Falling back to public interface 2021-07-26T06:07:40.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:41.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:41.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:41.591 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:07:41.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:07:41.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:41.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:07:41.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:42.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:42.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:42.864 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:07:42.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:07:42.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:42.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:07:42.995 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:07:42.996+0000 7f0b17f970c0 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T06:07:43.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:44.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:44.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:44.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:07:44.179 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:07:44.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:44.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:07:44.455 INFO:tasks.workunit.client.0.smithi071.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/33178,v1:127.0.0.1:6827/33178] [v2:127.0.0.1:6828/33178,v1:127.0.0.1:6829/33178] exists,up 4ed67cf2-6ba5-46b4-a188-c7f5346e051c 2021-07-26T06:07:44.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:07:44.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:07:44.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:07:44.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: create_pool test 1 1 2021-07-26T06:07:44.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-07-26T06:07:44.894 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'test' created 2021-07-26T06:07:44.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:07:45.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:58: TEST_recovery_scrub_1: wait_for_clean 2021-07-26T06:07:45.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:07:45.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:07:45.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:07:45.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:07:45.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:07:45.908 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:07:45.908 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:07:45.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:07:45.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:07:45.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:07:45.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:07:46.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:07:46.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:07:46.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:07:46.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:07:46.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:07:46.288 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:07:46.288 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:07:46.289 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T06:07:46.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:07:46.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:07:46.289 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:07:46.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836489 2021-07-26T06:07:46.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836489 2021-07-26T06:07:46.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489' 2021-07-26T06:07:46.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:07:46.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:07:46.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672967 2021-07-26T06:07:46.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672967 2021-07-26T06:07:46.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2021-07-26T06:07:46.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:07:46.560 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:07:46.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509444 2021-07-26T06:07:46.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509444 2021-07-26T06:07:46.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509444' 2021-07-26T06:07:46.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:07:46.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:07:46.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345922 2021-07-26T06:07:46.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345922 2021-07-26T06:07:46.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509444 3-85899345922' 2021-07-26T06:07:46.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:07:46.791 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836489 2021-07-26T06:07:46.792 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:07:46.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:07:46.792 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836489 2021-07-26T06:07:46.793 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:07:46.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836489 2021-07-26T06:07:46.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2021-07-26T06:07:46.794 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836489 2021-07-26T06:07:46.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:07:47.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836489 2021-07-26T06:07:47.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:07:47.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672967 2021-07-26T06:07:47.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:07:47.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:07:47.065 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672967 2021-07-26T06:07:47.066 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:07:47.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672967 2021-07-26T06:07:47.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2021-07-26T06:07:47.067 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672967 2021-07-26T06:07:47.068 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:07:47.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672967 -lt 42949672967 2021-07-26T06:07:47.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:07:47.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509444 2021-07-26T06:07:47.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:07:47.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:07:47.362 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509444 2021-07-26T06:07:47.362 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:07:47.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509444 2021-07-26T06:07:47.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-07-26T06:07:47.363 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509444 2021-07-26T06:07:47.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:07:47.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509444 -lt 64424509444 2021-07-26T06:07:47.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:07:47.672 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345922 2021-07-26T06:07:47.672 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:07:47.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:07:47.673 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345922 2021-07-26T06:07:47.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:07:47.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345922 2021-07-26T06:07:47.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2021-07-26T06:07:47.675 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.3 seq 85899345922 2021-07-26T06:07:47.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:07:47.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345922 -lt 85899345922 2021-07-26T06:07:47.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:07:47.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:07:47.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:07:48.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:07:48.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:07:48.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:07:48.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:07:48.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:07:48.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:07:48.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:07:48.298 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:07:48.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:07:48.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:07:48.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:07:48.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:07:48.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:07:48.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:07:48.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:07:48.954 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: ceph osd dump 2021-07-26T06:07:48.954 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2021-07-26T06:07:48.955 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: awk '{ print $2 }' 2021-07-26T06:07:49.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: poolid=1 2021-07-26T06:07:49.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: ceph pg dump pgs 2021-07-26T06:07:49.466 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:07:49.467 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:07:45.828582+0000 0'0 23:7 [1] 1 [1] 1 0'0 2021-07-26T06:07:44.799196+0000 0'0 2021-07-26T06:07:44.799196+0000 0 2021-07-26T06:07:49.467 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:07:49.468 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:07:49.469 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:07:49.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.27826 bs=1M count=50 2021-07-26T06:07:49.707 INFO:tasks.workunit.client.0.smithi071.stderr:50+0 records in 2021-07-26T06:07:49.707 INFO:tasks.workunit.client.0.smithi071.stderr:50+0 records out 2021-07-26T06:07:49.707 INFO:tasks.workunit.client.0.smithi071.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.228028 s, 230 MB/s 2021-07-26T06:07:49.708 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: seq 1 100 2021-07-26T06:07:49.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:07:49.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj1 testdata.27826 2021-07-26T06:07:51.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:07:51.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj2 testdata.27826 2021-07-26T06:07:53.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:07:53.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj3 testdata.27826 2021-07-26T06:07:54.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:07:54.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj4 testdata.27826 2021-07-26T06:07:57.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:07:57.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj5 testdata.27826 2021-07-26T06:08:00.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:00.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj6 testdata.27826 2021-07-26T06:08:01.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:01.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj7 testdata.27826 2021-07-26T06:08:03.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:03.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj8 testdata.27826 2021-07-26T06:08:05.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:05.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj9 testdata.27826 2021-07-26T06:08:07.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:07.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj10 testdata.27826 2021-07-26T06:08:10.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:10.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj11 testdata.27826 2021-07-26T06:08:11.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:11.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj12 testdata.27826 2021-07-26T06:08:14.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:14.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj13 testdata.27826 2021-07-26T06:08:16.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:16.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj14 testdata.27826 2021-07-26T06:08:18.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:18.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj15 testdata.27826 2021-07-26T06:08:20.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:20.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj16 testdata.27826 2021-07-26T06:08:22.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:22.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj17 testdata.27826 2021-07-26T06:08:24.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:24.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj18 testdata.27826 2021-07-26T06:08:26.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:26.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj19 testdata.27826 2021-07-26T06:08:29.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:29.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj20 testdata.27826 2021-07-26T06:08:31.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:31.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj21 testdata.27826 2021-07-26T06:08:34.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:34.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj22 testdata.27826 2021-07-26T06:08:35.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:35.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj23 testdata.27826 2021-07-26T06:08:38.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:38.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj24 testdata.27826 2021-07-26T06:08:40.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:40.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj25 testdata.27826 2021-07-26T06:08:42.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:42.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj26 testdata.27826 2021-07-26T06:08:44.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:44.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj27 testdata.27826 2021-07-26T06:08:46.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:46.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj28 testdata.27826 2021-07-26T06:08:48.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:48.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj29 testdata.27826 2021-07-26T06:08:50.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:50.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj30 testdata.27826 2021-07-26T06:08:52.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:52.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj31 testdata.27826 2021-07-26T06:08:54.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:54.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj32 testdata.27826 2021-07-26T06:08:56.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:56.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj33 testdata.27826 2021-07-26T06:08:58.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:08:58.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj34 testdata.27826 2021-07-26T06:09:00.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:00.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj35 testdata.27826 2021-07-26T06:09:02.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:02.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj36 testdata.27826 2021-07-26T06:09:04.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:04.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj37 testdata.27826 2021-07-26T06:09:08.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:08.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj38 testdata.27826 2021-07-26T06:09:10.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:10.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj39 testdata.27826 2021-07-26T06:09:13.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:13.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj40 testdata.27826 2021-07-26T06:09:14.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:14.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj41 testdata.27826 2021-07-26T06:09:17.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:17.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj42 testdata.27826 2021-07-26T06:09:20.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:20.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj43 testdata.27826 2021-07-26T06:09:22.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:22.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj44 testdata.27826 2021-07-26T06:09:24.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:24.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj45 testdata.27826 2021-07-26T06:09:26.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:26.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj46 testdata.27826 2021-07-26T06:09:29.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:29.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj47 testdata.27826 2021-07-26T06:09:31.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:31.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj48 testdata.27826 2021-07-26T06:09:33.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:33.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj49 testdata.27826 2021-07-26T06:09:35.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:35.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj50 testdata.27826 2021-07-26T06:09:37.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:37.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj51 testdata.27826 2021-07-26T06:09:41.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:41.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj52 testdata.27826 2021-07-26T06:09:43.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:43.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj53 testdata.27826 2021-07-26T06:09:45.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:45.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj54 testdata.27826 2021-07-26T06:09:46.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:46.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj55 testdata.27826 2021-07-26T06:09:49.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:49.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj56 testdata.27826 2021-07-26T06:09:51.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:51.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj57 testdata.27826 2021-07-26T06:09:54.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:54.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj58 testdata.27826 2021-07-26T06:09:55.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:55.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj59 testdata.27826 2021-07-26T06:09:57.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:57.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj60 testdata.27826 2021-07-26T06:09:59.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:09:59.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj61 testdata.27826 2021-07-26T06:10:02.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:02.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj62 testdata.27826 2021-07-26T06:10:04.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:04.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj63 testdata.27826 2021-07-26T06:10:06.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:06.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj64 testdata.27826 2021-07-26T06:10:09.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:09.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj65 testdata.27826 2021-07-26T06:10:12.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:12.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj66 testdata.27826 2021-07-26T06:10:14.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:14.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj67 testdata.27826 2021-07-26T06:10:16.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:16.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj68 testdata.27826 2021-07-26T06:10:18.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:18.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj69 testdata.27826 2021-07-26T06:10:21.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:21.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj70 testdata.27826 2021-07-26T06:10:24.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:24.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj71 testdata.27826 2021-07-26T06:10:26.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:26.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj72 testdata.27826 2021-07-26T06:10:28.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:28.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj73 testdata.27826 2021-07-26T06:10:30.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:30.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj74 testdata.27826 2021-07-26T06:10:33.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:33.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj75 testdata.27826 2021-07-26T06:10:35.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:35.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj76 testdata.27826 2021-07-26T06:10:37.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:37.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj77 testdata.27826 2021-07-26T06:10:39.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:39.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj78 testdata.27826 2021-07-26T06:10:42.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:42.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj79 testdata.27826 2021-07-26T06:10:44.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:44.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj80 testdata.27826 2021-07-26T06:10:46.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:46.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj81 testdata.27826 2021-07-26T06:10:48.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:48.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj82 testdata.27826 2021-07-26T06:10:51.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:51.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj83 testdata.27826 2021-07-26T06:10:54.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:54.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj84 testdata.27826 2021-07-26T06:10:56.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:56.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj85 testdata.27826 2021-07-26T06:10:59.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:10:59.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj86 testdata.27826 2021-07-26T06:11:01.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:01.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj87 testdata.27826 2021-07-26T06:11:03.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:03.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj88 testdata.27826 2021-07-26T06:11:06.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:06.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj89 testdata.27826 2021-07-26T06:11:07.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:07.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj90 testdata.27826 2021-07-26T06:11:10.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:10.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj91 testdata.27826 2021-07-26T06:11:11.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:11.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj92 testdata.27826 2021-07-26T06:11:14.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:14.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj93 testdata.27826 2021-07-26T06:11:16.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:16.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj94 testdata.27826 2021-07-26T06:11:18.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:18.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj95 testdata.27826 2021-07-26T06:11:20.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:20.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj96 testdata.27826 2021-07-26T06:11:22.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:22.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj97 testdata.27826 2021-07-26T06:11:24.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:24.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj98 testdata.27826 2021-07-26T06:11:26.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:26.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj99 testdata.27826 2021-07-26T06:11:28.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2021-07-26T06:11:28.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj100 testdata.27826 2021-07-26T06:11:30.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: rm -f testdata.27826 2021-07-26T06:11:30.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: ceph osd pool set test size 4 2021-07-26T06:11:30.756 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 1 size to 4 2021-07-26T06:11:30.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:73: TEST_recovery_scrub_1: set -o pipefail 2021-07-26T06:11:30.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: count=0 2021-07-26T06:11:30.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2021-07-26T06:11:30.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2021-07-26T06:11:30.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2021-07-26T06:11:30.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2021-07-26T06:11:31.181 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:11:31.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: sleep 2 2021-07-26T06:11:33.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:83: TEST_recovery_scrub_1: test 0 -eq 10 2021-07-26T06:11:33.193 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: expr 0 + 1 2021-07-26T06:11:33.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: count=1 2021-07-26T06:11:33.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2021-07-26T06:11:33.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2021-07-26T06:11:33.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2021-07-26T06:11:33.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2021-07-26T06:11:33.460 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:11:33.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: sleep 2 2021-07-26T06:11:35.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:83: TEST_recovery_scrub_1: test 1 -eq 10 2021-07-26T06:11:35.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: expr 1 + 1 2021-07-26T06:11:35.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: count=2 2021-07-26T06:11:35.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2021-07-26T06:11:35.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2021-07-26T06:11:35.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2021-07-26T06:11:35.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2021-07-26T06:11:35.948 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:11:35.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: sleep 2 2021-07-26T06:11:37.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:83: TEST_recovery_scrub_1: test 2 -eq 10 2021-07-26T06:11:37.959 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: expr 2 + 1 2021-07-26T06:11:37.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: count=3 2021-07-26T06:11:37.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2021-07-26T06:11:37.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2021-07-26T06:11:37.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2021-07-26T06:11:37.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2021-07-26T06:11:38.246 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:11:38.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:80: TEST_recovery_scrub_1: break 2021-07-26T06:11:38.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:90: TEST_recovery_scrub_1: set +o pipefail 2021-07-26T06:11:38.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:91: TEST_recovery_scrub_1: ceph pg dump pgs 2021-07-26T06:11:38.510 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:11:38.510 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 1300 active+recovering+undersized+degraded+remapped 2021-07-26T06:11:33.848895+0000 24'1300 27:1321 [1,0,2,3] 1 [1,0] 1 0'0 2021-07-26T06:07:44.799196+0000 0'0 2021-07-26T06:07:44.799196+0000 0 2021-07-26T06:11:38.511 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:11:38.511 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:11:38.512 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:11:38.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:93: TEST_recovery_scrub_1: sleep 10 2021-07-26T06:11:48.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2021-07-26T06:11:48.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:11:48.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:11:48.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:11:48.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:11:48.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:11:48.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:11:48.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: declare -a err_strings 2021-07-26T06:11:48.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:98: TEST_recovery_scrub_1: err_strings[0]='not scheduling scrubs due to active recovery' 2021-07-26T06:11:48.835 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: expr 4 - 1 2021-07-26T06:11:48.836 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: seq 0 3 2021-07-26T06:11:48.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:11:48.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2021-07-26T06:11:48.838 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:34.825+0000 7f9940d18700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.839 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:39.970+0000 7f9940d18700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.839 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:43.138+0000 7f9940d18700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.839 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:46.245+0000 7f9940d18700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.839 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:47.301+0000 7f9940d18700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:11:48.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2021-07-26T06:11:48.850 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:34.688+0000 7f117e64a700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.850 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:38.655+0000 7f117e64a700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.851 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:41.652+0000 7f117e64a700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.851 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:42.646+0000 7f117e64a700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.851 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:43.673+0000 7f117e64a700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.851 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:45.687+0000 7f117e64a700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.851 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:46.650+0000 7f117e64a700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.852 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:47.648+0000 7f117e64a700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:11:48.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2021-07-26T06:11:48.853 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:36.504+0000 7f965ee7b700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:11:48.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2021-07-26T06:11:48.855 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:40.503+0000 7f0b0d317700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.855 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:41.456+0000 7f0b0d317700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.855 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:43.468+0000 7f0b0d317700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.855 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:11:45.470+0000 7f0b0d317700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2021-07-26T06:11:48.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2021-07-26T06:11:48.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: found=false 2021-07-26T06:11:48.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:107: TEST_recovery_scrub_1: count=0 2021-07-26T06:11:48.857 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: expr 4 - 1 2021-07-26T06:11:48.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: seq 0 3 2021-07-26T06:11:48.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:11:48.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2021-07-26T06:11:48.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2021-07-26T06:11:48.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 0 + 1 2021-07-26T06:11:48.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=1 2021-07-26T06:11:48.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:11:48.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2021-07-26T06:11:48.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2021-07-26T06:11:48.870 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 1 + 1 2021-07-26T06:11:48.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=2 2021-07-26T06:11:48.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:11:48.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2021-07-26T06:11:48.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2021-07-26T06:11:48.873 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 2 + 1 2021-07-26T06:11:48.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=3 2021-07-26T06:11:48.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:11:48.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2021-07-26T06:11:48.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2021-07-26T06:11:48.876 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 3 + 1 2021-07-26T06:11:48.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=4 2021-07-26T06:11:48.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: '[' true = false ']' 2021-07-26T06:11:48.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2021-07-26T06:11:48.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:123: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2021-07-26T06:11:48.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2021-07-26T06:11:48.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:11:48.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2021-07-26T06:11:48.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:11:48.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:11:48.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:11:48.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:11:48.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:11:48.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:11:48.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:11:48.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:11:48.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:11:48.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:11:48.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:11:48.886 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:11:48.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:11:48.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:11:48.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:11:48.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:11:48.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:11:48.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:11:48.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2021-07-26T06:11:49.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:11:49.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:11:49.593 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:11:49.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27826 2021-07-26T06:11:49.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:11:49.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:11:49.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:125: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2021-07-26T06:11:49.595 INFO:tasks.workunit.client.0.smithi071.stdout:TEST PASSED 2021-07-26T06:11:49.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:131: TEST_recovery_scrub_1: echo 'TEST PASSED' 2021-07-26T06:11:49.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:132: TEST_recovery_scrub_1: return 0 2021-07-26T06:11:49.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: for func in $funcs 2021-07-26T06:11:49.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2021-07-26T06:11:49.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:230: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2021-07-26T06:11:49.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:231: TEST_recovery_scrub_2: local poolname=test 2021-07-26T06:11:49.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:233: TEST_recovery_scrub_2: TESTDATA=testdata.27826 2021-07-26T06:11:49.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: OSDS=8 2021-07-26T06:11:49.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: PGS=32 2021-07-26T06:11:49.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:236: TEST_recovery_scrub_2: OBJECTS=40 2021-07-26T06:11:49.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2021-07-26T06:11:49.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2021-07-26T06:11:49.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2021-07-26T06:11:49.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2021-07-26T06:11:49.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:11:49.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2021-07-26T06:11:49.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:11:49.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:11:49.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:11:49.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:11:49.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:11:49.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:11:49.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:11:49.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:11:49.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:11:49.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:11:49.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:11:49.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:11:49.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:11:49.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:11:49.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:11:49.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:11:49.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:11:49.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:11:49.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2021-07-26T06:11:49.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:11:49.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:11:49.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:11:49.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27826 2021-07-26T06:11:49.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:11:49.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:11:49.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2021-07-26T06:11:49.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:11:49.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:11:49.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:11:49.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27826 2021-07-26T06:11:49.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:11:49.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:11:49.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:11:49.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2021-07-26T06:11:49.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2021-07-26T06:11:49.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2021-07-26T06:11:49.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:11:49.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:11:49.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:11:49.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2021-07-26T06:11:49.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2021-07-26T06:11:49.987 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:11:49.987 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:11:49.987 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:11:49.988 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:11:49.988 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:11:49.988 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:11:49.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:11:49.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2021-07-26T06:11:50.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:11:50.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:11:50.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:11:50.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:11:50.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:11:50.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:11:50.291 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:11:50.291 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:11:50.292 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:11:50.292 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:11:50.292 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:11:50.292 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:11:50.293 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27826/ceph-mon.a.asok 2021-07-26T06:11:50.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:11:50.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.27826/ceph-mon.a.asok config get fsid 2021-07-26T06:11:50.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:11:50.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:11:50.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:11:50.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:11:50.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:11:50.389 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:11:50.389 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:11:50.389 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:11:50.389 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:11:50.390 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:11:50.390 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:11:50.390 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27826/ceph-mon.a.asok 2021-07-26T06:11:50.390 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:11:50.390 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.27826/ceph-mon.a.asok config get mon_host 2021-07-26T06:11:50.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:241: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2021-07-26T06:11:50.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2021-07-26T06:11:50.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:11:50.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:11:50.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:11:50.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2021-07-26T06:11:50.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:11:51.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:11:51.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:11:51.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:11:51.317 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:11:51.318 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:11:51.318 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:11:51.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:11:51.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:11:51.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:11:51.339 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: expr 8 - 1 2021-07-26T06:11:51.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: seq 0 7 2021-07-26T06:11:51.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:11:51.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:11:51.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2021-07-26T06:11:51.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:11:51.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:11:51.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:11:51.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2021-07-26T06:11:51.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2021-07-26T06:11:51.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:11:51.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:11:51.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:11:51.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2021-07-26T06:11:51.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2021-07-26T06:11:51.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:11:51.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:11:51.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:11:51.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2021-07-26T06:11:51.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:11:51.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:11:51.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:11:51.349 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:11:51.349 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:11:51.350 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:11:51.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:11:51.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:11:51.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:11:51.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:11:51.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:11:51.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2021-07-26T06:11:51.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2021-07-26T06:11:51.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:11:51.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:11:51.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:11:51.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:11:51.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2021-07-26T06:11:51.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2021-07-26T06:11:51.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:11:51.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cfed5f44-4c2c-4572-8dcb-63235555e76a 2021-07-26T06:11:51.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cfed5f44-4c2c-4572-8dcb-63235555e76a' 2021-07-26T06:11:51.353 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 cfed5f44-4c2c-4572-8dcb-63235555e76a 2021-07-26T06:11:51.354 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:11:51.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAnUv5gDN1dFRAAksbAKhYqd9ZATsmu0s/ebw== 2021-07-26T06:11:51.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAnUv5gDN1dFRAAksbAKhYqd9ZATsmu0s/ebw=="}' 2021-07-26T06:11:51.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cfed5f44-4c2c-4572-8dcb-63235555e76a -i td/osd-recovery-scrub/0/new.json 2021-07-26T06:11:51.606 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:11:51.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2021-07-26T06:11:51.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQAnUv5gDN1dFRAAksbAKhYqd9ZATsmu0s/ebw== --osd-uuid cfed5f44-4c2c-4572-8dcb-63235555e76a 2021-07-26T06:11:51.640 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:11:51.640+0000 7f62cf1d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:51.641 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:11:51.641+0000 7f62cf1d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:51.642 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:11:51.642+0000 7f62cf1d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:51.643 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:11:51.642+0000 7f62cf1d80c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2021-07-26T06:11:51.643 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:11:51.642+0000 7f62cf1d80c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2021-07-26T06:11:51.643 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:11:51.642+0000 7f62cf1d80c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2021-07-26T06:11:51.723 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:11:51.723+0000 7f62cf1d80c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2021-07-26T06:11:57.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2021-07-26T06:11:57.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:11:57.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:11:57.530 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:11:57.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:11:57.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:11:57.884 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:11:57.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:11:57.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:11:57.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:11:57.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:11:57.906 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:11:57.906+0000 7fee1d5d90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:57.907 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:11:57.907+0000 7fee1d5d90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:57.908 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:11:57.908+0000 7fee1d5d90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:58.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:11:58.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:11:58.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:11:58.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:11:58.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:11:58.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:11:58.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:11:58.211 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:11:58.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:11:58.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:11:58.423 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:11:58.424+0000 7fee1d5d90c0 -1 Falling back to public interface 2021-07-26T06:11:58.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:11:59.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:11:59.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:11:59.520 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:11:59.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:11:59.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:11:59.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:11:59.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:00.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:00.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:00.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:12:00.792 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:12:00.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:00.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:12:00.833 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:00.834+0000 7fee1d5d90c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:12:01.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:02.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:02.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:02.141 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:12:02.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:12:02.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:02.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:12:02.414 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/37619,v1:127.0.0.1:6803/37619] [v2:127.0.0.1:6804/37619,v1:127.0.0.1:6805/37619] exists,up cfed5f44-4c2c-4572-8dcb-63235555e76a 2021-07-26T06:12:02.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:12:02.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:12:02.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:12:02.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:12:02.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:12:02.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2021-07-26T06:12:02.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:12:02.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:12:02.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:12:02.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2021-07-26T06:12:02.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2021-07-26T06:12:02.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:12:02.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:12:02.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:12:02.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2021-07-26T06:12:02.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2021-07-26T06:12:02.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:12:02.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:12:02.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:12:02.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2021-07-26T06:12:02.425 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:12:02.425 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:12:02.425 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:12:02.425 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:12:02.426 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:12:02.426 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:12:02.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:12:02.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:12:02.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:12:02.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:12:02.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:12:02.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2021-07-26T06:12:02.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2021-07-26T06:12:02.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:12:02.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:12:02.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:12:02.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:12:02.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2021-07-26T06:12:02.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2021-07-26T06:12:02.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:12:02.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e4f72068-e503-4271-8402-4ea22f72e01d 2021-07-26T06:12:02.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e4f72068-e503-4271-8402-4ea22f72e01d' 2021-07-26T06:12:02.430 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 e4f72068-e503-4271-8402-4ea22f72e01d 2021-07-26T06:12:02.431 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:12:02.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAyUv5gjrO6GRAAHra6e8iPeuFgvL/cizNFyQ== 2021-07-26T06:12:02.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAyUv5gjrO6GRAAHra6e8iPeuFgvL/cizNFyQ=="}' 2021-07-26T06:12:02.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e4f72068-e503-4271-8402-4ea22f72e01d -i td/osd-recovery-scrub/1/new.json 2021-07-26T06:12:02.802 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:12:02.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2021-07-26T06:12:02.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQAyUv5gjrO6GRAAHra6e8iPeuFgvL/cizNFyQ== --osd-uuid e4f72068-e503-4271-8402-4ea22f72e01d 2021-07-26T06:12:02.837 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:02.838+0000 7f2f5207c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:02.855 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:02.852+0000 7f2f5207c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:02.857 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:02.858+0000 7f2f5207c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:02.857 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:02.858+0000 7f2f5207c0c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2021-07-26T06:12:02.858 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:02.858+0000 7f2f5207c0c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2021-07-26T06:12:02.858 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:02.858+0000 7f2f5207c0c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2021-07-26T06:12:02.962 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:02.963+0000 7f2f5207c0c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2021-07-26T06:12:08.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2021-07-26T06:12:08.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:12:08.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:12:08.662 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:12:08.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:12:08.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:12:08.999 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:12:09.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:12:09.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:12:09.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:12:09.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:12:09.020 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:09.021+0000 7f9f2c7600c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:09.021 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:09.023+0000 7f9f2c7600c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:09.022 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:09.023+0000 7f9f2c7600c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:09.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:12:09.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:12:09.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:12:09.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:12:09.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:12:09.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:09.298 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:12:09.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:12:09.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:09.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:12:09.545 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:09.547+0000 7f9f2c7600c0 -1 Falling back to public interface 2021-07-26T06:12:09.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:10.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:10.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:10.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:12:10.605 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:12:10.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:10.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:12:10.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:11.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:11.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:11.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:12:11.881 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:12:11.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:11.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:12:11.928 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:11.929+0000 7f9f2c7600c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:12:12.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:13.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:13.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:13.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:12:13.195 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:12:13.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:13.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:12:13.462 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/39018,v1:127.0.0.1:6811/39018] [v2:127.0.0.1:6812/39018,v1:127.0.0.1:6813/39018] exists,up e4f72068-e503-4271-8402-4ea22f72e01d 2021-07-26T06:12:13.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:12:13.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:12:13.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:12:13.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:12:13.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:12:13.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2021-07-26T06:12:13.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:12:13.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:12:13.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:12:13.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2021-07-26T06:12:13.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2021-07-26T06:12:13.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:12:13.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:12:13.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:12:13.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2021-07-26T06:12:13.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2021-07-26T06:12:13.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:12:13.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:12:13.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:12:13.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2021-07-26T06:12:13.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:12:13.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:12:13.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:12:13.473 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:12:13.473 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:12:13.473 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:12:13.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:12:13.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:12:13.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:12:13.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:12:13.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:12:13.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2021-07-26T06:12:13.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2021-07-26T06:12:13.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:12:13.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:12:13.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:12:13.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:12:13.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2021-07-26T06:12:13.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2021-07-26T06:12:13.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:12:13.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43b02f4f-83f7-4df0-bae0-9ae75646e5bb 2021-07-26T06:12:13.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 43b02f4f-83f7-4df0-bae0-9ae75646e5bb' 2021-07-26T06:12:13.478 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 43b02f4f-83f7-4df0-bae0-9ae75646e5bb 2021-07-26T06:12:13.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:12:13.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9Uv5g1RWUHBAABAhMfkNO+FriI8eMac+5QA== 2021-07-26T06:12:13.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9Uv5g1RWUHBAABAhMfkNO+FriI8eMac+5QA=="}' 2021-07-26T06:12:13.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43b02f4f-83f7-4df0-bae0-9ae75646e5bb -i td/osd-recovery-scrub/2/new.json 2021-07-26T06:12:13.828 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:12:13.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2021-07-26T06:12:13.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQA9Uv5g1RWUHBAABAhMfkNO+FriI8eMac+5QA== --osd-uuid 43b02f4f-83f7-4df0-bae0-9ae75646e5bb 2021-07-26T06:12:13.863 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:13.864+0000 7f263a1af0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:13.880 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:13.881+0000 7f263a1af0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:13.881 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:13.882+0000 7f263a1af0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:13.881 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:13.882+0000 7f263a1af0c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2021-07-26T06:12:13.882 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:13.882+0000 7f263a1af0c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2021-07-26T06:12:13.882 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:13.882+0000 7f263a1af0c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2021-07-26T06:12:13.981 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:13.982+0000 7f263a1af0c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2021-07-26T06:12:19.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2021-07-26T06:12:19.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:12:19.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:12:19.974 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:12:19.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:12:20.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:12:20.311 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:12:20.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:12:20.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:12:20.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:12:20.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:12:20.331 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:20.332+0000 7f09987a60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:20.333 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:20.334+0000 7f09987a60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:20.334 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:20.335+0000 7f09987a60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:20.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:12:20.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:12:20.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:12:20.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:12:20.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:12:20.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:20.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:12:20.577 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:12:20.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:20.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:12:20.851 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:20.851+0000 7f09987a60c0 -1 Falling back to public interface 2021-07-26T06:12:20.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:21.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:21.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:21.890 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:12:21.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:12:21.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:21.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:12:22.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:23.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:23.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:23.162 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:12:23.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:12:23.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:23.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:12:23.252 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:23.253+0000 7f09987a60c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:12:23.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:24.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:24.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:24.494 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:12:24.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:12:24.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:24.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:12:24.768 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/40425,v1:127.0.0.1:6819/40425] [v2:127.0.0.1:6820/40425,v1:127.0.0.1:6821/40425] exists,up 43b02f4f-83f7-4df0-bae0-9ae75646e5bb 2021-07-26T06:12:24.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:12:24.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:12:24.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:12:24.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:12:24.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:12:24.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2021-07-26T06:12:24.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:12:24.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T06:12:24.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:12:24.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2021-07-26T06:12:24.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2021-07-26T06:12:24.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:12:24.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:12:24.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:12:24.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2021-07-26T06:12:24.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2021-07-26T06:12:24.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:12:24.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:12:24.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:12:24.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2021-07-26T06:12:24.777 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:12:24.778 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:12:24.778 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:12:24.778 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:12:24.778 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:12:24.779 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:12:24.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:12:24.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:12:24.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:12:24.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:12:24.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:12:24.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2021-07-26T06:12:24.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2021-07-26T06:12:24.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:12:24.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:12:24.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:12:24.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:12:24.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2021-07-26T06:12:24.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2021-07-26T06:12:24.782 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:12:24.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=783c63dd-1df9-4f18-a52d-0f1ffe7ed14f 2021-07-26T06:12:24.782 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 783c63dd-1df9-4f18-a52d-0f1ffe7ed14f 2021-07-26T06:12:24.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 783c63dd-1df9-4f18-a52d-0f1ffe7ed14f' 2021-07-26T06:12:24.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:12:24.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIUv5gbEbRLhAA96OPgJefacc5KBVCm6FO4A== 2021-07-26T06:12:24.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIUv5gbEbRLhAA96OPgJefacc5KBVCm6FO4A=="}' 2021-07-26T06:12:24.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 783c63dd-1df9-4f18-a52d-0f1ffe7ed14f -i td/osd-recovery-scrub/3/new.json 2021-07-26T06:12:25.197 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:12:25.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2021-07-26T06:12:25.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQBIUv5gbEbRLhAA96OPgJefacc5KBVCm6FO4A== --osd-uuid 783c63dd-1df9-4f18-a52d-0f1ffe7ed14f 2021-07-26T06:12:25.231 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:25.233+0000 7f319888d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:25.246 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:25.247+0000 7f319888d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:25.247 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:25.248+0000 7f319888d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:25.247 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:25.248+0000 7f319888d0c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2021-07-26T06:12:25.248 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:25.248+0000 7f319888d0c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2021-07-26T06:12:25.248 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:25.248+0000 7f319888d0c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2021-07-26T06:12:25.317 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:25.318+0000 7f319888d0c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2021-07-26T06:12:31.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2021-07-26T06:12:31.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:12:31.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T06:12:31.735 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd3 key to auth repository 2021-07-26T06:12:31.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:12:32.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T06:12:32.078 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.3 2021-07-26T06:12:32.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:12:32.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:12:32.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:12:32.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:12:32.099 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:32.100+0000 7f4b73e410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:32.101 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:32.102+0000 7f4b73e410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:32.101 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:32.103+0000 7f4b73e410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:32.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T06:12:32.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:12:32.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:12:32.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:12:32.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:12:32.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:32.388 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:12:32.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:12:32.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:32.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:12:32.617 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:32.618+0000 7f4b73e410c0 -1 Falling back to public interface 2021-07-26T06:12:32.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:33.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:33.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:33.692 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:12:33.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:12:33.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:33.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:12:33.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:34.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:34.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:34.969 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:12:34.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:12:34.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:34.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:12:34.989 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:34.990+0000 7f4b73e410c0 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T06:12:35.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:36.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:36.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:36.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:12:36.253 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:12:36.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:36.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:12:36.518 INFO:tasks.workunit.client.0.smithi071.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/41824,v1:127.0.0.1:6827/41824] [v2:127.0.0.1:6828/41824,v1:127.0.0.1:6829/41824] exists,up 783c63dd-1df9-4f18-a52d-0f1ffe7ed14f 2021-07-26T06:12:36.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:12:36.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:12:36.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:12:36.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:12:36.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:12:36.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2021-07-26T06:12:36.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:12:36.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-07-26T06:12:36.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:12:36.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2021-07-26T06:12:36.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2021-07-26T06:12:36.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:12:36.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:12:36.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:12:36.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2021-07-26T06:12:36.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2021-07-26T06:12:36.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:12:36.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:12:36.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:12:36.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2021-07-26T06:12:36.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:12:36.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:12:36.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:12:36.529 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:12:36.529 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:12:36.529 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:12:36.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:12:36.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:12:36.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:12:36.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:12:36.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:12:36.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2021-07-26T06:12:36.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2021-07-26T06:12:36.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:12:36.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:12:36.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:12:36.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:12:36.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2021-07-26T06:12:36.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2021-07-26T06:12:36.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:12:36.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0983f1ac-744f-4ef0-82f3-c1dbcdd85fff 2021-07-26T06:12:36.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 0983f1ac-744f-4ef0-82f3-c1dbcdd85fff' 2021-07-26T06:12:36.533 INFO:tasks.workunit.client.0.smithi071.stdout:add osd4 0983f1ac-744f-4ef0-82f3-c1dbcdd85fff 2021-07-26T06:12:36.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:12:36.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBUUv5gHUL0HxAA2OT+bNzGyA9d5OXoOGpdVg== 2021-07-26T06:12:36.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBUUv5gHUL0HxAA2OT+bNzGyA9d5OXoOGpdVg=="}' 2021-07-26T06:12:36.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0983f1ac-744f-4ef0-82f3-c1dbcdd85fff -i td/osd-recovery-scrub/4/new.json 2021-07-26T06:12:36.959 INFO:tasks.workunit.client.0.smithi071.stdout:4 2021-07-26T06:12:36.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2021-07-26T06:12:36.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQBUUv5gHUL0HxAA2OT+bNzGyA9d5OXoOGpdVg== --osd-uuid 0983f1ac-744f-4ef0-82f3-c1dbcdd85fff 2021-07-26T06:12:36.994 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:36.994+0000 7fd4a1d9c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:37.014 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:37.014+0000 7fd4a1d9c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:37.015 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:37.015+0000 7fd4a1d9c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:37.016 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:37.016+0000 7fd4a1d9c0c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2021-07-26T06:12:37.016 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:37.016+0000 7fd4a1d9c0c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2021-07-26T06:12:37.016 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:37.016+0000 7fd4a1d9c0c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2021-07-26T06:12:37.092 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:37.092+0000 7fd4a1d9c0c0 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2021-07-26T06:12:43.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2021-07-26T06:12:43.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:12:43.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2021-07-26T06:12:43.105 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd4 key to auth repository 2021-07-26T06:12:43.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:12:43.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2021-07-26T06:12:43.444 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.4 2021-07-26T06:12:43.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:12:43.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:12:43.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:12:43.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:12:43.464 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:43.464+0000 7f71fa5a50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:43.466 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:43.466+0000 7f71fa5a50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:43.467 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:43.467+0000 7f71fa5a50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:43.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2021-07-26T06:12:43.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:12:43.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=4 2021-07-26T06:12:43.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:12:43.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:12:43.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:43.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:12:43.714 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:12:43.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:43.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:12:43.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:43.986 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:43.986+0000 7f71fa5a50c0 -1 Falling back to public interface 2021-07-26T06:12:44.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:44.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:44.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:12:44.977 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:12:44.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:44.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:12:45.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:46.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:46.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:46.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:12:46.250 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:12:46.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:46.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:12:46.392 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:46.392+0000 7f71fa5a50c0 -1 osd.4 0 log_to_monitors {default=true} 2021-07-26T06:12:46.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:47.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:47.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:47.550 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:12:47.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:12:47.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:47.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:12:47.858 INFO:tasks.workunit.client.0.smithi071.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/43223,v1:127.0.0.1:6835/43223] [v2:127.0.0.1:6836/43223,v1:127.0.0.1:6837/43223] exists,up 0983f1ac-744f-4ef0-82f3-c1dbcdd85fff 2021-07-26T06:12:47.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:12:47.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:12:47.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:12:47.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:12:47.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:12:47.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2021-07-26T06:12:47.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:12:47.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2021-07-26T06:12:47.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:12:47.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2021-07-26T06:12:47.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2021-07-26T06:12:47.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:12:47.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:12:47.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:12:47.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2021-07-26T06:12:47.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2021-07-26T06:12:47.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:12:47.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:12:47.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:12:47.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2021-07-26T06:12:47.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:12:47.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:12:47.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:12:47.867 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:12:47.868 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:12:47.868 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:12:47.868 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:12:47.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:12:47.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:12:47.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:12:47.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:12:47.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2021-07-26T06:12:47.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2021-07-26T06:12:47.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:12:47.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:12:47.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:12:47.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:12:47.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2021-07-26T06:12:47.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2021-07-26T06:12:47.871 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:12:47.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=91ab4b57-5f26-435a-a693-23e3266ed287 2021-07-26T06:12:47.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 91ab4b57-5f26-435a-a693-23e3266ed287' 2021-07-26T06:12:47.871 INFO:tasks.workunit.client.0.smithi071.stdout:add osd5 91ab4b57-5f26-435a-a693-23e3266ed287 2021-07-26T06:12:47.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:12:47.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfUv5gXD04NBAAETnREQvc5GKFqPqCPpa4Qg== 2021-07-26T06:12:47.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfUv5gXD04NBAAETnREQvc5GKFqPqCPpa4Qg=="}' 2021-07-26T06:12:47.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 91ab4b57-5f26-435a-a693-23e3266ed287 -i td/osd-recovery-scrub/5/new.json 2021-07-26T06:12:48.388 INFO:tasks.workunit.client.0.smithi071.stdout:5 2021-07-26T06:12:48.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2021-07-26T06:12:48.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQBfUv5gXD04NBAAETnREQvc5GKFqPqCPpa4Qg== --osd-uuid 91ab4b57-5f26-435a-a693-23e3266ed287 2021-07-26T06:12:48.423 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:48.423+0000 7f7b2ed510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:48.446 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:48.446+0000 7f7b2ed510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:48.447 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:48.447+0000 7f7b2ed510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:48.448 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:48.447+0000 7f7b2ed510c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2021-07-26T06:12:48.448 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:48.447+0000 7f7b2ed510c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2021-07-26T06:12:48.448 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:48.447+0000 7f7b2ed510c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2021-07-26T06:12:48.521 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:48.521+0000 7f7b2ed510c0 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2021-07-26T06:12:54.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2021-07-26T06:12:54.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:12:54.720 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd5 key to auth repository 2021-07-26T06:12:54.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2021-07-26T06:12:54.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:12:55.060 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.5 2021-07-26T06:12:55.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2021-07-26T06:12:55.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:12:55.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:12:55.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:12:55.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:12:55.081 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:55.081+0000 7efcb8f2b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:55.083 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:55.083+0000 7efcb8f2b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:55.084 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:55.084+0000 7efcb8f2b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:55.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2021-07-26T06:12:55.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:12:55.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:12:55.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:12:55.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:12:55.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:55.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:12:55.329 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:12:55.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:55.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:12:55.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:55.611 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:55.612+0000 7efcb8f2b0c0 -1 Falling back to public interface 2021-07-26T06:12:56.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:56.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:56.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:12:56.593 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:12:56.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:56.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:12:56.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:57.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:57.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:57.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:12:57.860 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:12:57.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:57.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:12:58.074 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:58.074+0000 7efcb8f2b0c0 -1 osd.5 0 log_to_monitors {default=true} 2021-07-26T06:12:58.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:59.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:59.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:59.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:12:59.149 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:12:59.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:59.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:12:59.419 INFO:tasks.workunit.client.0.smithi071.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/44622,v1:127.0.0.1:6843/44622] [v2:127.0.0.1:6844/44622,v1:127.0.0.1:6845/44622] exists,up 91ab4b57-5f26-435a-a693-23e3266ed287 2021-07-26T06:12:59.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:12:59.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:12:59.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:12:59.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:12:59.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:12:59.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2021-07-26T06:12:59.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:12:59.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2021-07-26T06:12:59.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:12:59.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2021-07-26T06:12:59.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2021-07-26T06:12:59.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:12:59.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:12:59.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:12:59.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2021-07-26T06:12:59.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2021-07-26T06:12:59.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:12:59.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:12:59.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:12:59.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2021-07-26T06:12:59.428 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:12:59.428 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:12:59.428 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:12:59.429 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:12:59.429 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:12:59.429 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:12:59.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:12:59.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:12:59.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:12:59.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:12:59.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:12:59.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2021-07-26T06:12:59.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2021-07-26T06:12:59.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:12:59.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:12:59.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:12:59.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:12:59.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2021-07-26T06:12:59.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2021-07-26T06:12:59.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:12:59.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7acb833a-5e47-430b-9c26-3947b6aff2ba 2021-07-26T06:12:59.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 7acb833a-5e47-430b-9c26-3947b6aff2ba' 2021-07-26T06:12:59.433 INFO:tasks.workunit.client.0.smithi071.stdout:add osd6 7acb833a-5e47-430b-9c26-3947b6aff2ba 2021-07-26T06:12:59.433 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:12:59.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBrUv5gPv0BGhAAYTDSRULxwTYRGEHT3+60og== 2021-07-26T06:12:59.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBrUv5gPv0BGhAAYTDSRULxwTYRGEHT3+60og=="}' 2021-07-26T06:12:59.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7acb833a-5e47-430b-9c26-3947b6aff2ba -i td/osd-recovery-scrub/6/new.json 2021-07-26T06:12:59.819 INFO:tasks.workunit.client.0.smithi071.stdout:6 2021-07-26T06:12:59.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2021-07-26T06:12:59.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQBrUv5gPv0BGhAAYTDSRULxwTYRGEHT3+60og== --osd-uuid 7acb833a-5e47-430b-9c26-3947b6aff2ba 2021-07-26T06:12:59.854 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:59.855+0000 7f50701d90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:59.856 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:59.857+0000 7f50701d90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:59.856 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:59.858+0000 7f50701d90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:59.857 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:59.858+0000 7f50701d90c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2021-07-26T06:12:59.858 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:59.858+0000 7f50701d90c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2021-07-26T06:12:59.858 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:59.858+0000 7f50701d90c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2021-07-26T06:12:59.935 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:12:59.936+0000 7f50701d90c0 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2021-07-26T06:13:06.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2021-07-26T06:13:06.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:13:06.215 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd6 key to auth repository 2021-07-26T06:13:06.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2021-07-26T06:13:06.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:13:06.556 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.6 2021-07-26T06:13:06.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2021-07-26T06:13:06.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:13:06.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:13:06.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:13:06.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:13:06.577 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:06.578+0000 7fa56adb80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:13:06.579 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:06.580+0000 7fa56adb80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:13:06.580 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:06.581+0000 7fa56adb80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:13:06.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2021-07-26T06:13:06.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:13:06.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=6 2021-07-26T06:13:06.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:13:06.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:13:06.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:13:06.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:13:06.825 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:13:06.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:13:06.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.6 up' 2021-07-26T06:13:07.101 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:07.102+0000 7fa56adb80c0 -1 Falling back to public interface 2021-07-26T06:13:07.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:13:08.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:13:08.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:13:08.130 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:13:08.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:13:08.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:13:08.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.6 up' 2021-07-26T06:13:08.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:13:09.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:13:09.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:13:09.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:13:09.407 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:13:09.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:13:09.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.6 up' 2021-07-26T06:13:09.498 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:09.499+0000 7fa56adb80c0 -1 osd.6 0 log_to_monitors {default=true} 2021-07-26T06:13:09.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:13:10.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:13:10.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:13:10.707 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:13:10.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:13:10.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:13:10.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.6 up' 2021-07-26T06:13:10.985 INFO:tasks.workunit.client.0.smithi071.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/46024,v1:127.0.0.1:6851/46024] [v2:127.0.0.1:6852/46024,v1:127.0.0.1:6853/46024] exists,up 7acb833a-5e47-430b-9c26-3947b6aff2ba 2021-07-26T06:13:10.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:13:10.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:13:10.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:13:10.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:13:10.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:13:10.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2021-07-26T06:13:10.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:13:10.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2021-07-26T06:13:10.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:13:10.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2021-07-26T06:13:10.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2021-07-26T06:13:10.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:13:10.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:13:10.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:13:10.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2021-07-26T06:13:10.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2021-07-26T06:13:10.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:13:10.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:13:10.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:13:10.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2021-07-26T06:13:10.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:13:10.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:13:10.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:13:10.998 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:13:10.998 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:13:10.998 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:13:10.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:13:10.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' 2021-07-26T06:13:10.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:13:10.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:13:10.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:13:10.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2021-07-26T06:13:11.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2021-07-26T06:13:11.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:13:11.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:13:11.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:13:11.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:13:11.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2021-07-26T06:13:11.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2021-07-26T06:13:11.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:13:11.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=88d51c06-2ed6-4d49-a914-de52ba95a444 2021-07-26T06:13:11.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 88d51c06-2ed6-4d49-a914-de52ba95a444' 2021-07-26T06:13:11.002 INFO:tasks.workunit.client.0.smithi071.stdout:add osd7 88d51c06-2ed6-4d49-a914-de52ba95a444 2021-07-26T06:13:11.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:13:11.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3Uv5g6TwsABAAm/Mg7pzaULJG4YB2EVnrGA== 2021-07-26T06:13:11.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3Uv5g6TwsABAAm/Mg7pzaULJG4YB2EVnrGA=="}' 2021-07-26T06:13:11.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 88d51c06-2ed6-4d49-a914-de52ba95a444 -i td/osd-recovery-scrub/7/new.json 2021-07-26T06:13:11.343 INFO:tasks.workunit.client.0.smithi071.stdout:7 2021-07-26T06:13:11.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2021-07-26T06:13:11.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQB3Uv5g6TwsABAAm/Mg7pzaULJG4YB2EVnrGA== --osd-uuid 88d51c06-2ed6-4d49-a914-de52ba95a444 2021-07-26T06:13:11.379 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:11.380+0000 7feb324e50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:13:11.413 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:11.411+0000 7feb324e50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:13:11.414 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:11.415+0000 7feb324e50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:13:11.415 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:11.416+0000 7feb324e50c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2021-07-26T06:13:11.415 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:11.416+0000 7feb324e50c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2021-07-26T06:13:11.415 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:11.416+0000 7feb324e50c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2021-07-26T06:13:11.510 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:11.511+0000 7feb324e50c0 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2021-07-26T06:13:17.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2021-07-26T06:13:17.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:13:17.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2021-07-26T06:13:17.591 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd7 key to auth repository 2021-07-26T06:13:17.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:13:17.945 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.7 2021-07-26T06:13:17.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2021-07-26T06:13:17.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=866e5c93-c59c-4765-9d95-f188d73ebc4f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.27826/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2021-07-26T06:13:17.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:13:17.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:13:17.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:13:17.967 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:17.968+0000 7fd50d6fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:13:17.969 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:17.970+0000 7fd50d6fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:13:17.970 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:17.971+0000 7fd50d6fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:13:18.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2021-07-26T06:13:18.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:13:18.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=7 2021-07-26T06:13:18.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:13:18.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:13:18.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:13:18.250 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:13:18.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:13:18.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:13:18.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.7 up' 2021-07-26T06:13:18.491 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:18.492+0000 7fd50d6fc0c0 -1 Falling back to public interface 2021-07-26T06:13:18.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:13:19.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:13:19.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:13:19.516 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:13:19.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:13:19.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:13:19.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.7 up' 2021-07-26T06:13:19.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:13:20.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:13:20.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:13:20.794 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:13:20.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:13:20.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:13:20.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.7 up' 2021-07-26T06:13:21.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:13:21.079 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:13:21.080+0000 7fd50d6fc0c0 -1 osd.7 0 log_to_monitors {default=true} 2021-07-26T06:13:22.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:13:22.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:13:22.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:13:22.073 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:13:22.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:13:22.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.7 up' 2021-07-26T06:13:22.345 INFO:tasks.workunit.client.0.smithi071.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/47431,v1:127.0.0.1:6859/47431] [v2:127.0.0.1:6860/47431,v1:127.0.0.1:6861/47431] exists,up 88d51c06-2ed6-4d49-a914-de52ba95a444 2021-07-26T06:13:22.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:13:22.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:13:22.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:13:22.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: create_pool test 32 32 2021-07-26T06:13:22.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2021-07-26T06:13:22.828 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'test' created 2021-07-26T06:13:22.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:13:23.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: wait_for_clean 2021-07-26T06:13:23.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:13:23.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:13:23.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:13:23.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:13:23.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:13:23.842 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:13:23.843 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:13:23.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:13:23.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:13:23.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:13:23.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:13:23.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:13:23.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:13:23.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:13:23.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:13:24.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:13:24.365 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:13:24.365 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:13:24.366 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:13:24.366 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:13:24.366 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:13:24.366 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:13:24.367 INFO:tasks.workunit.client.0.smithi071.stderr:7' 2021-07-26T06:13:24.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:13:24.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:13:24.368 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:13:24.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836498 2021-07-26T06:13:24.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836498 2021-07-26T06:13:24.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498' 2021-07-26T06:13:24.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:13:24.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:13:25.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672976 2021-07-26T06:13:25.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672976 2021-07-26T06:13:25.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2021-07-26T06:13:25.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:13:25.053 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:13:25.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509455 2021-07-26T06:13:25.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509455 2021-07-26T06:13:25.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509455' 2021-07-26T06:13:25.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:13:25.260 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:13:25.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345931 2021-07-26T06:13:25.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345931 2021-07-26T06:13:25.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509455 3-85899345931' 2021-07-26T06:13:25.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:13:25.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:13:25.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182409 2021-07-26T06:13:25.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182409 2021-07-26T06:13:25.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509455 3-85899345931 4-107374182409' 2021-07-26T06:13:25.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:13:25.512 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:13:25.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018888 2021-07-26T06:13:25.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018888 2021-07-26T06:13:25.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509455 3-85899345931 4-107374182409 5-128849018888' 2021-07-26T06:13:25.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:13:25.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:13:25.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855364 2021-07-26T06:13:25.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855364 2021-07-26T06:13:25.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509455 3-85899345931 4-107374182409 5-128849018888 6-150323855364' 2021-07-26T06:13:25.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:13:25.747 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:13:25.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691842 2021-07-26T06:13:25.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691842 2021-07-26T06:13:25.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509455 3-85899345931 4-107374182409 5-128849018888 6-150323855364 7-171798691842' 2021-07-26T06:13:25.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:13:25.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836498 2021-07-26T06:13:25.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:13:25.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:13:25.865 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836498 2021-07-26T06:13:25.865 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:13:25.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836498 2021-07-26T06:13:25.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2021-07-26T06:13:25.866 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836498 2021-07-26T06:13:25.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:13:26.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836498 -lt 21474836498 2021-07-26T06:13:26.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:13:26.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672976 2021-07-26T06:13:26.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:13:26.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:13:26.136 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672976 2021-07-26T06:13:26.136 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:13:26.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672976 2021-07-26T06:13:26.137 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672976 2021-07-26T06:13:26.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2021-07-26T06:13:26.138 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:13:26.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672976 -lt 42949672976 2021-07-26T06:13:26.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:13:26.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509455 2021-07-26T06:13:26.405 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:13:26.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:13:26.406 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509455 2021-07-26T06:13:26.406 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:13:26.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509455 2021-07-26T06:13:26.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2021-07-26T06:13:26.408 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509455 2021-07-26T06:13:26.408 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:13:26.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509455 -lt 64424509455 2021-07-26T06:13:26.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:13:26.713 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345931 2021-07-26T06:13:26.713 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:13:26.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:13:26.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345931 2021-07-26T06:13:26.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:13:26.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345931 2021-07-26T06:13:26.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2021-07-26T06:13:26.716 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.3 seq 85899345931 2021-07-26T06:13:26.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:13:26.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345931 -lt 85899345931 2021-07-26T06:13:26.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:13:26.982 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182409 2021-07-26T06:13:26.982 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:13:26.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:13:26.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182409 2021-07-26T06:13:26.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:13:26.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182409 2021-07-26T06:13:26.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2021-07-26T06:13:26.984 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.4 seq 107374182409 2021-07-26T06:13:26.985 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:13:27.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182409 -lt 107374182409 2021-07-26T06:13:27.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:13:27.325 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018888 2021-07-26T06:13:27.326 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:13:27.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:13:27.328 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018888 2021-07-26T06:13:27.329 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:13:27.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018888 2021-07-26T06:13:27.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2021-07-26T06:13:27.330 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.5 seq 128849018888 2021-07-26T06:13:27.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:13:27.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018888 -lt 128849018888 2021-07-26T06:13:27.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:13:27.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855364 2021-07-26T06:13:27.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:13:27.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:13:27.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855364 2021-07-26T06:13:27.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:13:27.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855364 2021-07-26T06:13:27.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855364' 2021-07-26T06:13:27.635 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.6 seq 150323855364 2021-07-26T06:13:27.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:13:27.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855364 -lt 150323855364 2021-07-26T06:13:27.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:13:27.899 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691842 2021-07-26T06:13:27.899 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:13:27.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:13:27.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691842 2021-07-26T06:13:27.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:13:27.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691842 2021-07-26T06:13:27.902 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.7 seq 171798691842 2021-07-26T06:13:27.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2021-07-26T06:13:27.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:13:28.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691843 -lt 171798691842 2021-07-26T06:13:28.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:13:28.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:28.169 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:28.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 32 == 0 2021-07-26T06:13:28.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:13:28.564 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:13:28.564 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:13:28.564 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:13:28.564 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:13:28.565 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:13:28.565 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:13:28.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=32 2021-07-26T06:13:28.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:13:28.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:28.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:29.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 32 = 32 2021-07-26T06:13:29.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:13:29.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:13:29.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:250: TEST_recovery_scrub_2: ceph osd dump 2021-07-26T06:13:29.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:250: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2021-07-26T06:13:29.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:250: TEST_recovery_scrub_2: awk '{ print $2 }' 2021-07-26T06:13:29.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:250: TEST_recovery_scrub_2: poolid=1 2021-07-26T06:13:29.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:252: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.27826 bs=1M count=50 2021-07-26T06:13:29.685 INFO:tasks.workunit.client.0.smithi071.stderr:50+0 records in 2021-07-26T06:13:29.685 INFO:tasks.workunit.client.0.smithi071.stderr:50+0 records out 2021-07-26T06:13:29.685 INFO:tasks.workunit.client.0.smithi071.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.231366 s, 227 MB/s 2021-07-26T06:13:29.686 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: seq 1 40 2021-07-26T06:13:29.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:29.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj1 testdata.27826 2021-07-26T06:13:31.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:31.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj2 testdata.27826 2021-07-26T06:13:33.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:33.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj3 testdata.27826 2021-07-26T06:13:35.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:35.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj4 testdata.27826 2021-07-26T06:13:37.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:37.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj5 testdata.27826 2021-07-26T06:13:39.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:39.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj6 testdata.27826 2021-07-26T06:13:41.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:41.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj7 testdata.27826 2021-07-26T06:13:43.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:43.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj8 testdata.27826 2021-07-26T06:13:45.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:45.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj9 testdata.27826 2021-07-26T06:13:47.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:47.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj10 testdata.27826 2021-07-26T06:13:49.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:49.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj11 testdata.27826 2021-07-26T06:13:51.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:51.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj12 testdata.27826 2021-07-26T06:13:53.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:53.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj13 testdata.27826 2021-07-26T06:13:56.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:56.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj14 testdata.27826 2021-07-26T06:13:58.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:58.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj15 testdata.27826 2021-07-26T06:13:59.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:13:59.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj16 testdata.27826 2021-07-26T06:14:02.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:02.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj17 testdata.27826 2021-07-26T06:14:04.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:04.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj18 testdata.27826 2021-07-26T06:14:06.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:06.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj19 testdata.27826 2021-07-26T06:14:08.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:08.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj20 testdata.27826 2021-07-26T06:14:10.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:10.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj21 testdata.27826 2021-07-26T06:14:13.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:13.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj22 testdata.27826 2021-07-26T06:14:15.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:15.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj23 testdata.27826 2021-07-26T06:14:17.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:17.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj24 testdata.27826 2021-07-26T06:14:19.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:19.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj25 testdata.27826 2021-07-26T06:14:22.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:22.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj26 testdata.27826 2021-07-26T06:14:23.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:23.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj27 testdata.27826 2021-07-26T06:14:26.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:26.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj28 testdata.27826 2021-07-26T06:14:27.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:27.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj29 testdata.27826 2021-07-26T06:14:30.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:30.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj30 testdata.27826 2021-07-26T06:14:33.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:33.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj31 testdata.27826 2021-07-26T06:14:35.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:35.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj32 testdata.27826 2021-07-26T06:14:36.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:36.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj33 testdata.27826 2021-07-26T06:14:39.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj34 testdata.27826 2021-07-26T06:14:41.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:41.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj35 testdata.27826 2021-07-26T06:14:43.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:43.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj36 testdata.27826 2021-07-26T06:14:44.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:44.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj37 testdata.27826 2021-07-26T06:14:47.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:47.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj38 testdata.27826 2021-07-26T06:14:49.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:49.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj39 testdata.27826 2021-07-26T06:14:51.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2021-07-26T06:14:51.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj40 testdata.27826 2021-07-26T06:14:53.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: rm -f testdata.27826 2021-07-26T06:14:53.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:259: TEST_recovery_scrub_2: ceph osd pool set test size 3 2021-07-26T06:14:53.800 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 1 size to 3 2021-07-26T06:14:53.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: ceph pg dump pgs 2021-07-26T06:14:54.064 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:14:54.064 INFO:tasks.workunit.client.0.smithi071.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 26 active+clean 2021-07-26T06:13:24.185388+0000 46'26 46:37 [5] 5 [5] 5 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.064 INFO:tasks.workunit.client.0.smithi071.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.662915+0000 46'13 46:24 [3] 3 [3] 3 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.065 INFO:tasks.workunit.client.0.smithi071.stdout:1.1d 2 0 0 0 0 102760448 0 0 25 25 active+clean 2021-07-26T06:13:23.690314+0000 46'25 46:35 [1] 1 [1] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.065 INFO:tasks.workunit.client.0.smithi071.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.662827+0000 46'13 46:23 [0] 0 [0] 0 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.065 INFO:tasks.workunit.client.0.smithi071.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.663187+0000 46'13 46:23 [0] 0 [0] 0 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.065 INFO:tasks.workunit.client.0.smithi071.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:13:23.690169+0000 0'0 44:10 [1] 1 [1] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.066 INFO:tasks.workunit.client.0.smithi071.stdout:1.19 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.662799+0000 46'13 46:24 [3] 3 [3] 3 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.066 INFO:tasks.workunit.client.0.smithi071.stdout:1.18 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:13:23.991594+0000 0'0 45:11 [5] 5 [5] 5 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.066 INFO:tasks.workunit.client.0.smithi071.stdout:1.17 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.662706+0000 46'13 46:23 [0] 0 [0] 0 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.066 INFO:tasks.workunit.client.0.smithi071.stdout:1.16 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:13:24.791220+0000 0'0 44:10 [6] 6 [6] 6 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.067 INFO:tasks.workunit.client.0.smithi071.stdout:1.15 3 0 0 0 0 157286400 0 0 39 39 active+clean 2021-07-26T06:13:23.597781+0000 46'39 46:49 [1] 1 [1] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.067 INFO:tasks.workunit.client.0.smithi071.stdout:1.14 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:13:24.628281+0000 0'0 44:10 [2] 2 [2] 2 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.067 INFO:tasks.workunit.client.0.smithi071.stdout:1.13 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.662483+0000 46'13 46:23 [4] 4 [4] 4 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.067 INFO:tasks.workunit.client.0.smithi071.stdout:1.12 2 0 0 0 0 104857600 0 0 26 26 active+clean 2021-07-26T06:13:24.743159+0000 46'26 46:36 [0] 0 [0] 0 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.068 INFO:tasks.workunit.client.0.smithi071.stdout:1.11 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.662914+0000 46'13 46:23 [0] 0 [0] 0 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.068 INFO:tasks.workunit.client.0.smithi071.stdout:1.10 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.185161+0000 46'13 46:24 [5] 5 [5] 5 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.068 INFO:tasks.workunit.client.0.smithi071.stdout:1.5 3 0 0 0 0 157286400 0 0 39 39 active+clean 2021-07-26T06:13:24.743198+0000 46'39 46:49 [4] 4 [4] 4 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.068 INFO:tasks.workunit.client.0.smithi071.stdout:1.4 4 0 0 0 0 209715200 0 0 52 52 active+clean 2021-07-26T06:13:24.662988+0000 46'52 46:63 [3] 3 [3] 3 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.069 INFO:tasks.workunit.client.0.smithi071.stdout:1.3 4 0 0 0 0 209715200 0 0 52 52 active+clean 2021-07-26T06:13:24.743248+0000 46'52 46:62 [4] 4 [4] 4 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.069 INFO:tasks.workunit.client.0.smithi071.stdout:1.2 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.663036+0000 46'13 46:23 [0] 0 [0] 0 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.069 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.662531+0000 46'13 46:22 [7] 7 [7] 7 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.070 INFO:tasks.workunit.client.0.smithi071.stdout:1.1 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.185280+0000 46'13 46:24 [5] 5 [5] 5 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.070 INFO:tasks.workunit.client.0.smithi071.stdout:1.6 1 0 0 0 0 37748736 0 0 9 9 active+clean 2021-07-26T06:13:24.628282+0000 46'9 46:19 [6] 6 [6] 6 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.071 INFO:tasks.workunit.client.0.smithi071.stdout:1.7 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.791156+0000 46'13 46:23 [6] 6 [6] 6 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.071 INFO:tasks.workunit.client.0.smithi071.stdout:1.8 2 0 0 0 0 104857600 0 0 26 26 active+clean 2021-07-26T06:13:24.184477+0000 46'26 46:37 [5] 5 [5] 5 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.072 INFO:tasks.workunit.client.0.smithi071.stdout:1.9 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:13:23.690235+0000 0'0 44:10 [1] 1 [1] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.072 INFO:tasks.workunit.client.0.smithi071.stdout:1.a 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:23.597823+0000 46'13 46:23 [1] 1 [1] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.072 INFO:tasks.workunit.client.0.smithi071.stdout:1.b 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:23.690281+0000 46'13 46:23 [1] 1 [1] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.072 INFO:tasks.workunit.client.0.smithi071.stdout:1.c 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:13:24.662969+0000 0'0 45:11 [3] 3 [3] 3 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.073 INFO:tasks.workunit.client.0.smithi071.stdout:1.d 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.743145+0000 46'13 46:22 [7] 7 [7] 7 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.073 INFO:tasks.workunit.client.0.smithi071.stdout:1.e 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:23.689993+0000 46'13 46:23 [1] 1 [1] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.073 INFO:tasks.workunit.client.0.smithi071.stdout:1.f 1 0 0 0 0 52428800 0 0 13 13 active+clean 2021-07-26T06:13:24.790962+0000 46'13 46:23 [6] 6 [6] 6 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:14:54.073 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:14:54.074 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:14:54.074 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:14:54.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:264: TEST_recovery_scrub_2: count=0 2021-07-26T06:14:54.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: true 2021-07-26T06:14:54.077 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2021-07-26T06:14:54.077 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2021-07-26T06:14:54.077 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: grep recovering 2021-07-26T06:14:54.077 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: wc -l 2021-07-26T06:14:54.428 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:14:54.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: test 0 -ge 2 2021-07-26T06:14:54.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:273: TEST_recovery_scrub_2: sleep 2 2021-07-26T06:14:56.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:274: TEST_recovery_scrub_2: test 0 -eq 10 2021-07-26T06:14:56.440 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:279: TEST_recovery_scrub_2: expr 0 + 1 2021-07-26T06:14:56.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:279: TEST_recovery_scrub_2: count=1 2021-07-26T06:14:56.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: true 2021-07-26T06:14:56.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2021-07-26T06:14:56.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2021-07-26T06:14:56.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: grep recovering 2021-07-26T06:14:56.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: wc -l 2021-07-26T06:14:56.730 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:14:56.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: test 0 -ge 2 2021-07-26T06:14:56.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:273: TEST_recovery_scrub_2: sleep 2 2021-07-26T06:14:58.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:274: TEST_recovery_scrub_2: test 1 -eq 10 2021-07-26T06:14:58.742 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:279: TEST_recovery_scrub_2: expr 1 + 1 2021-07-26T06:14:58.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:279: TEST_recovery_scrub_2: count=2 2021-07-26T06:14:58.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: true 2021-07-26T06:14:58.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2021-07-26T06:14:58.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2021-07-26T06:14:58.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: grep recovering 2021-07-26T06:14:58.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: wc -l 2021-07-26T06:14:59.007 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:14:59.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: test 1 -ge 2 2021-07-26T06:14:59.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:273: TEST_recovery_scrub_2: sleep 2 2021-07-26T06:15:01.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:274: TEST_recovery_scrub_2: test 2 -eq 10 2021-07-26T06:15:01.019 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:279: TEST_recovery_scrub_2: expr 2 + 1 2021-07-26T06:15:01.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:279: TEST_recovery_scrub_2: count=3 2021-07-26T06:15:01.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: true 2021-07-26T06:15:01.021 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2021-07-26T06:15:01.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2021-07-26T06:15:01.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: grep recovering 2021-07-26T06:15:01.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: wc -l 2021-07-26T06:15:01.285 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:15:01.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: test 2 -ge 2 2021-07-26T06:15:01.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:271: TEST_recovery_scrub_2: break 2021-07-26T06:15:01.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: ceph pg dump pgs 2021-07-26T06:15:01.544 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:15:01.545 INFO:tasks.workunit.client.0.smithi071.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 26 active+recovery_wait+degraded 2021-07-26T06:14:56.286554+0000 46'26 49:47 [5,0,2] 5 [5,0,2] 5 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.545 INFO:tasks.workunit.client.0.smithi071.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 13 activating+degraded 2021-07-26T06:14:56.361535+0000 46'13 49:29 [3,0,7] 3 [3,0,7] 3 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.545 INFO:tasks.workunit.client.0.smithi071.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 26 active+recovery_wait+degraded 2021-07-26T06:14:56.505102+0000 46'26 49:45 [1,3,0] 1 [1,3,0] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.546 INFO:tasks.workunit.client.0.smithi071.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 13 active+recovery_wait+degraded 2021-07-26T06:14:56.362949+0000 46'13 49:33 [0,3,4] 0 [0,3,4] 0 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.546 INFO:tasks.workunit.client.0.smithi071.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 13 active+recovery_wait+degraded 2021-07-26T06:14:56.722714+0000 46'13 49:33 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.546 INFO:tasks.workunit.client.0.smithi071.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:14:56.169927+0000 0'0 49:21 [1,7,4] 1 [1,7,4] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.546 INFO:tasks.workunit.client.0.smithi071.stdout:1.19 1 0 2 0 0 52428800 0 0 13 13 activating+degraded 2021-07-26T06:14:56.361852+0000 46'13 49:29 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.547 INFO:tasks.workunit.client.0.smithi071.stdout:1.18 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:14:56.169096+0000 0'0 49:23 [5,2,3] 5 [5,2,3] 5 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.547 INFO:tasks.workunit.client.0.smithi071.stdout:1.17 1 0 2 0 0 52428800 0 0 13 13 active+recovery_wait+degraded 2021-07-26T06:14:56.597599+0000 46'13 49:33 [0,4,5] 0 [0,4,5] 0 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.547 INFO:tasks.workunit.client.0.smithi071.stdout:1.16 0 0 0 0 0 0 0 0 0 0 activating 2021-07-26T06:14:55.748641+0000 0'0 49:14 [6,0,7] 6 [6,0,7] 6 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.547 INFO:tasks.workunit.client.0.smithi071.stdout:1.15 3 0 6 0 0 157286400 0 0 39 39 active+recovery_wait+degraded 2021-07-26T06:14:56.596681+0000 46'39 49:58 [1,5,3] 1 [1,5,3] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.548 INFO:tasks.workunit.client.0.smithi071.stdout:1.14 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:14:56.948290+0000 0'0 49:22 [2,0,4] 2 [2,0,4] 2 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.548 INFO:tasks.workunit.client.0.smithi071.stdout:1.13 1 0 2 0 0 52428800 0 0 13 13 active+recovery_wait+degraded 2021-07-26T06:14:56.596757+0000 46'13 49:32 [4,6,3] 4 [4,6,3] 4 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.548 INFO:tasks.workunit.client.0.smithi071.stdout:1.12 2 0 4 0 0 104857600 0 0 26 26 active+recovery_wait+degraded 2021-07-26T06:14:56.596975+0000 46'26 49:46 [0,1,6] 0 [0,1,6] 0 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.548 INFO:tasks.workunit.client.0.smithi071.stdout:1.11 1 0 2 0 0 52428800 0 0 13 13 active+recovery_wait+degraded 2021-07-26T06:14:56.806124+0000 46'13 49:33 [0,1,3] 0 [0,1,3] 0 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.548 INFO:tasks.workunit.client.0.smithi071.stdout:1.10 1 0 2 0 0 52428800 0 0 13 13 active+recovering+degraded 2021-07-26T06:14:56.170741+0000 46'13 49:35 [5,7,6] 5 [5,7,6] 5 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.549 INFO:tasks.workunit.client.0.smithi071.stdout:1.5 3 0 6 0 0 157286400 0 0 39 39 active+recovery_wait+degraded 2021-07-26T06:14:57.122347+0000 46'39 49:58 [4,2,6] 4 [4,2,6] 4 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.549 INFO:tasks.workunit.client.0.smithi071.stdout:1.4 4 0 8 0 0 209715200 0 0 52 52 activating+degraded 2021-07-26T06:14:56.361823+0000 46'52 49:68 [3,0,6] 3 [3,0,6] 3 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.549 INFO:tasks.workunit.client.0.smithi071.stdout:1.3 4 0 8 0 0 209715200 0 0 52 52 active+recovery_wait+degraded 2021-07-26T06:14:56.722391+0000 46'52 49:71 [4,1,2] 4 [4,1,2] 4 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.549 INFO:tasks.workunit.client.0.smithi071.stdout:1.2 1 0 2 0 0 52428800 0 0 13 13 active+recovery_wait+degraded 2021-07-26T06:14:56.596898+0000 46'13 49:33 [0,7,1] 0 [0,7,1] 0 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.550 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 1 0 2 0 0 52428800 0 0 13 13 active+recovery_wait+degraded 2021-07-26T06:14:57.284269+0000 46'13 49:32 [7,0,2] 7 [7,0,2] 7 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.550 INFO:tasks.workunit.client.0.smithi071.stdout:1.1 1 0 2 0 0 52428800 0 0 13 13 active+recovery_wait+degraded 2021-07-26T06:14:56.169986+0000 46'13 49:34 [5,0,4] 5 [5,0,4] 5 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.550 INFO:tasks.workunit.client.0.smithi071.stdout:1.6 1 0 2 0 0 52428800 0 0 13 13 activating 2021-07-26T06:14:55.751228+0000 46'13 49:27 [6,7,3] 6 [6,7,3] 6 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.551 INFO:tasks.workunit.client.0.smithi071.stdout:1.7 1 0 2 0 0 52428800 0 0 13 13 activating 2021-07-26T06:14:55.749254+0000 46'13 49:27 [6,7,3] 6 [6,7,3] 6 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.551 INFO:tasks.workunit.client.0.smithi071.stdout:1.8 2 0 4 0 0 104857600 0 0 26 26 active+recovery_wait+degraded 2021-07-26T06:14:56.362254+0000 46'26 49:47 [5,4,3] 5 [5,4,3] 5 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.551 INFO:tasks.workunit.client.0.smithi071.stdout:1.9 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:14:56.339801+0000 0'0 49:21 [1,4,5] 1 [1,4,5] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.551 INFO:tasks.workunit.client.0.smithi071.stdout:1.a 1 0 2 0 0 52428800 0 0 13 13 active+recovery_wait+degraded 2021-07-26T06:14:56.949196+0000 46'13 49:32 [1,6,2] 1 [1,6,2] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.552 INFO:tasks.workunit.client.0.smithi071.stdout:1.b 1 0 2 0 0 52428800 0 0 13 13 active+recovery_wait+degraded 2021-07-26T06:14:56.723031+0000 46'13 49:32 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.552 INFO:tasks.workunit.client.0.smithi071.stdout:1.c 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:14:56.286031+0000 0'0 49:21 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.552 INFO:tasks.workunit.client.0.smithi071.stdout:1.d 1 0 2 0 0 52428800 0 0 13 13 active+recovering+degraded 2021-07-26T06:14:56.949593+0000 46'13 49:33 [7,4,2] 7 [7,4,2] 7 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.552 INFO:tasks.workunit.client.0.smithi071.stdout:1.e 1 0 2 0 0 52428800 0 0 13 13 active+recovery_wait+degraded 2021-07-26T06:14:56.722391+0000 46'13 49:32 [1,4,2] 1 [1,4,2] 1 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.553 INFO:tasks.workunit.client.0.smithi071.stdout:1.f 1 0 2 0 0 52428800 0 0 13 13 activating 2021-07-26T06:14:55.749598+0000 46'13 49:27 [6,0,3] 6 [6,0,3] 6 0'0 2021-07-26T06:13:22.659593+0000 0'0 2021-07-26T06:13:22.659593+0000 0 2021-07-26T06:15:01.553 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:15:01.553 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:15:01.553 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:15:01.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:283: TEST_recovery_scrub_2: pids= 2021-07-26T06:15:01.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:284: TEST_recovery_scrub_2: recov_scrub_count=0 2021-07-26T06:15:01.557 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: expr 32 - 1 2021-07-26T06:15:01.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: seq 0 31 2021-07-26T06:15:01.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.562 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 0 2021-07-26T06:15:01.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2021-07-26T06:15:01.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49660"' 2021-07-26T06:15:01.563 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49660' 2021-07-26T06:15:01.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.0 2021-07-26T06:15:01.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49663: /' 2021-07-26T06:15:01.564 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 1 2021-07-26T06:15:01.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2021-07-26T06:15:01.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49666"' 2021-07-26T06:15:01.565 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49666' 2021-07-26T06:15:01.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.1 2021-07-26T06:15:01.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 2 2021-07-26T06:15:01.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49670: /' 2021-07-26T06:15:01.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2021-07-26T06:15:01.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49672"' 2021-07-26T06:15:01.567 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49672' 2021-07-26T06:15:01.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.2 2021-07-26T06:15:01.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 3 2021-07-26T06:15:01.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49677: /' 2021-07-26T06:15:01.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2021-07-26T06:15:01.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49680"' 2021-07-26T06:15:01.569 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49680' 2021-07-26T06:15:01.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.572 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 4 2021-07-26T06:15:01.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2021-07-26T06:15:01.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.3 2021-07-26T06:15:01.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49685"' 2021-07-26T06:15:01.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49685' 2021-07-26T06:15:01.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49684: /' 2021-07-26T06:15:01.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49689: /' 2021-07-26T06:15:01.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.4 2021-07-26T06:15:01.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 5 2021-07-26T06:15:01.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2021-07-26T06:15:01.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49695"' 2021-07-26T06:15:01.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49695' 2021-07-26T06:15:01.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.5 2021-07-26T06:15:01.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49698: /' 2021-07-26T06:15:01.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 6 2021-07-26T06:15:01.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2021-07-26T06:15:01.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49702"' 2021-07-26T06:15:01.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49702' 2021-07-26T06:15:01.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 7 2021-07-26T06:15:01.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2021-07-26T06:15:01.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49704"' 2021-07-26T06:15:01.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49704' 2021-07-26T06:15:01.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.583 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 8 2021-07-26T06:15:01.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2021-07-26T06:15:01.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49706"' 2021-07-26T06:15:01.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49706' 2021-07-26T06:15:01.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 9 2021-07-26T06:15:01.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49709: /' 2021-07-26T06:15:01.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2021-07-26T06:15:01.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49710"' 2021-07-26T06:15:01.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49710' 2021-07-26T06:15:01.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.8 2021-07-26T06:15:01.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 10 2021-07-26T06:15:01.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2021-07-26T06:15:01.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49715"' 2021-07-26T06:15:01.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49715' 2021-07-26T06:15:01.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 11 2021-07-26T06:15:01.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49718: /' 2021-07-26T06:15:01.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2021-07-26T06:15:01.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49719"' 2021-07-26T06:15:01.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49719' 2021-07-26T06:15:01.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 12 2021-07-26T06:15:01.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2021-07-26T06:15:01.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49722"' 2021-07-26T06:15:01.593 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49722' 2021-07-26T06:15:01.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.a 2021-07-26T06:15:01.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 13 2021-07-26T06:15:01.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2021-07-26T06:15:01.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49726"' 2021-07-26T06:15:01.595 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49726' 2021-07-26T06:15:01.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49723: /' 2021-07-26T06:15:01.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 14 2021-07-26T06:15:01.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2021-07-26T06:15:01.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49730"' 2021-07-26T06:15:01.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49730' 2021-07-26T06:15:01.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.6 2021-07-26T06:15:01.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49733: /' 2021-07-26T06:15:01.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.7 2021-07-26T06:15:01.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49741: /' 2021-07-26T06:15:01.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.9 2021-07-26T06:15:01.601 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 15 2021-07-26T06:15:01.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.b 2021-07-26T06:15:01.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2021-07-26T06:15:01.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49749"' 2021-07-26T06:15:01.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49749' 2021-07-26T06:15:01.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49736: /' 2021-07-26T06:15:01.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49747: /' 2021-07-26T06:15:01.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 16 2021-07-26T06:15:01.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.e 2021-07-26T06:15:01.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.d 2021-07-26T06:15:01.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49746: /' 2021-07-26T06:15:01.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49743: /' 2021-07-26T06:15:01.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.c 2021-07-26T06:15:01.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2021-07-26T06:15:01.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49763"' 2021-07-26T06:15:01.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49763' 2021-07-26T06:15:01.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49753: /' 2021-07-26T06:15:01.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.f 2021-07-26T06:15:01.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 17 2021-07-26T06:15:01.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2021-07-26T06:15:01.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49770"' 2021-07-26T06:15:01.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49770' 2021-07-26T06:15:01.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.10 2021-07-26T06:15:01.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 18 2021-07-26T06:15:01.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2021-07-26T06:15:01.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49781"' 2021-07-26T06:15:01.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49781' 2021-07-26T06:15:01.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49773: /' 2021-07-26T06:15:01.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 19 2021-07-26T06:15:01.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2021-07-26T06:15:01.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49783"' 2021-07-26T06:15:01.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49783' 2021-07-26T06:15:01.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49780: /' 2021-07-26T06:15:01.643 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 20 2021-07-26T06:15:01.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2021-07-26T06:15:01.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49791"' 2021-07-26T06:15:01.645 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49791' 2021-07-26T06:15:01.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49790: /' 2021-07-26T06:15:01.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.11 2021-07-26T06:15:01.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49786: /' 2021-07-26T06:15:01.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.13 2021-07-26T06:15:01.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 21 2021-07-26T06:15:01.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2021-07-26T06:15:01.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49795"' 2021-07-26T06:15:01.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49795' 2021-07-26T06:15:01.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.12 2021-07-26T06:15:01.659 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 22 2021-07-26T06:15:01.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2021-07-26T06:15:01.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49800"' 2021-07-26T06:15:01.660 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49800' 2021-07-26T06:15:01.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.14 2021-07-26T06:15:01.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 23 2021-07-26T06:15:01.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2021-07-26T06:15:01.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49811"' 2021-07-26T06:15:01.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49811' 2021-07-26T06:15:01.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.15 2021-07-26T06:15:01.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 24 2021-07-26T06:15:01.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2021-07-26T06:15:01.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49816"' 2021-07-26T06:15:01.675 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49816' 2021-07-26T06:15:01.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.16 2021-07-26T06:15:01.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49799: /' 2021-07-26T06:15:01.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.17 2021-07-26T06:15:01.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49814: /' 2021-07-26T06:15:01.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 25 2021-07-26T06:15:01.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2021-07-26T06:15:01.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49826"' 2021-07-26T06:15:01.686 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49826' 2021-07-26T06:15:01.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 26 2021-07-26T06:15:01.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49821: /' 2021-07-26T06:15:01.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49803: /' 2021-07-26T06:15:01.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.18 2021-07-26T06:15:01.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2021-07-26T06:15:01.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49824: /' 2021-07-26T06:15:01.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49837"' 2021-07-26T06:15:01.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49837' 2021-07-26T06:15:01.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49836: /' 2021-07-26T06:15:01.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.19 2021-07-26T06:15:01.710 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 27 2021-07-26T06:15:01.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2021-07-26T06:15:01.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49845"' 2021-07-26T06:15:01.717 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49845' 2021-07-26T06:15:01.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49841: /' 2021-07-26T06:15:01.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 28 2021-07-26T06:15:01.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2021-07-26T06:15:01.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49848"' 2021-07-26T06:15:01.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49848' 2021-07-26T06:15:01.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.1a 2021-07-26T06:15:01.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 29 2021-07-26T06:15:01.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49851: /' 2021-07-26T06:15:01.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.1b 2021-07-26T06:15:01.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2021-07-26T06:15:01.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49856"' 2021-07-26T06:15:01.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49856' 2021-07-26T06:15:01.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49854: /' 2021-07-26T06:15:01.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.1c 2021-07-26T06:15:01.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 30 2021-07-26T06:15:01.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2021-07-26T06:15:01.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49868"' 2021-07-26T06:15:01.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49868' 2021-07-26T06:15:01.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2021-07-26T06:15:01.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.1d 2021-07-26T06:15:01.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49863: /' 2021-07-26T06:15:01.788 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 31 2021-07-26T06:15:01.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49873: /' 2021-07-26T06:15:01.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2021-07-26T06:15:01.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:15:01.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:15:01.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 49876"' 2021-07-26T06:15:01.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 49876' 2021-07-26T06:15:01.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:289: TEST_recovery_scrub_2: wait_background_check pids 2021-07-26T06:15:01.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:206: wait_background_check: pids=' 49660 49666 49672 49680 49685 49695 49702 49704 49706 49710 49715 49719 49722 49726 49730 49749 49763 49770 49781 49783 49791 49795 49800 49811 49816 49826 49837 49845 49848 49856 49868 49876' 2021-07-26T06:15:01.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:208: wait_background_check: return_code=0 2021-07-26T06:15:01.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:01.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49660 2021-07-26T06:15:01.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.1e 2021-07-26T06:15:01.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/49879: /' 2021-07-26T06:15:01.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: pg_scrub_mod 1.1f 2021-07-26T06:15:08.255 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.10 2021-07-26T06:15:08.255 INFO:tasks.workunit.client.0.smithi071.stderr:49773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.10 2021-07-26T06:15:08.256 INFO:tasks.workunit.client.0.smithi071.stderr:49773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.10 2021-07-26T06:15:08.256 INFO:tasks.workunit.client.0.smithi071.stderr:49773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:08.256 INFO:tasks.workunit.client.0.smithi071.stderr:49773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2021-07-26T06:15:08.256 INFO:tasks.workunit.client.0.smithi071.stderr:49773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:08.257 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:08.257 INFO:tasks.workunit.client.0.smithi071.stderr:49773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:08.257 INFO:tasks.workunit.client.0.smithi071.stderr:49773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.10 query 2021-07-26T06:15:08.257 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:08.258 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:08.258 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:08.258 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.10 2021-07-26T06:15:08.258 INFO:tasks.workunit.client.0.smithi071.stderr:49773: instructing pg 1.10 on osd.5 to scrub 2021-07-26T06:15:08.258 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:08.259 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2021-07-26T06:15:08.259 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:08.259 INFO:tasks.workunit.client.0.smithi071.stderr:49773: dumped pgs 2021-07-26T06:15:08.259 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: recovery=true 2021-07-26T06:15:08.260 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.10 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:08.260 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.10 2021-07-26T06:15:08.260 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:08.260 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:08.261 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:08.261 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:08.261 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:08.261 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:08.261 INFO:tasks.workunit.client.0.smithi071.stderr:49773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2021-07-26T06:15:08.262 INFO:tasks.workunit.client.0.smithi071.stderr:49773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.10 2021-07-26T06:15:08.262 INFO:tasks.workunit.client.0.smithi071.stderr:49773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:08.262 INFO:tasks.workunit.client.0.smithi071.stderr:49773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2021-07-26T06:15:09.035 INFO:tasks.workunit.client.0.smithi071.stderr:49773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: g49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.17 2021-07-26T06:15:09.035 INFO:tasks.workunit.client.0.smithi071.stderr:49821: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.17 2021-07-26T06:15:09.035 INFO:tasks.workunit.client.0.smithi071.stderr:49821: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.17 2021-07-26T06:15:09.036 INFO:tasks.workunit.client.0.smithi071.stderr:49821: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.036 INFO:tasks.workunit.client.0.smithi071.stderr:49821: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2021-07-26T06:15:09.036 INFO:tasks.workunit.client.0.smithi071.stderr:49821: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.036 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.037 INFO:tasks.workunit.client.0.smithi071.stderr:49821: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.17 query 2021-07-26T06:15:09.037 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.037 INFO:tasks.workunit.client.0.smithi071.stderr:49821: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.037 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.038 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.038 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.17 2021-07-26T06:15:09.038 INFO:tasks.workunit.client.0.smithi071.stderr:49821: instructing pg 1.17 on osd.0 to scrub 2021-07-26T06:15:09.038 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2021-07-26T06:15:09.039 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.039 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.039 INFO:tasks.workunit.client.0.smithi071.stderr:49821: dumped pgs 2021-07-26T06:15:09.039 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.17 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.039 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.17 2021-07-26T06:15:09.040 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.040 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.040 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.040 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.041 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.041 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.041 INFO:tasks.workunit.client.0.smithi071.stderr:49821: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2021-07-26T06:15:09.041 INFO:tasks.workunit.client.0.smithi071.stderr:49821: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.17 2021-07-26T06:15:09.041 INFO:tasks.workunit.client.0.smithi071.stderr:49821: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.042 INFO:tasks.workunit.client.0.smithi071.stderr:49821: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2021-07-26T06:15:09.042 INFO:tasks.workunit.client.0.smithi071.stderr:49821: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.042 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.042 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.043 INFO:tasks.workunit.client.0.smithi071.stderr:49821: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.223 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.4 2021-07-26T06:15:09.223 INFO:tasks.workunit.client.0.smithi071.stderr:49689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.4 2021-07-26T06:15:09.224 INFO:tasks.workunit.client.0.smithi071.stderr:49689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.4 2021-07-26T06:15:09.224 INFO:tasks.workunit.client.0.smithi071.stderr:49689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.224 INFO:tasks.workunit.client.0.smithi071.stderr:49689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.224 INFO:tasks.workunit.client.0.smithi071.stderr:49689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2021-07-26T06:15:09.225 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.225 INFO:tasks.workunit.client.0.smithi071.stderr:49689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.4 query 2021-07-26T06:15:09.225 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.225 INFO:tasks.workunit.client.0.smithi071.stderr:49689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.225 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.226 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.226 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.4 2021-07-26T06:15:09.226 INFO:tasks.workunit.client.0.smithi071.stderr:49689: instructing pg 1.4 on osd.3 to scrub 2021-07-26T06:15:09.226 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.227 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2021-07-26T06:15:09.227 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.227 INFO:tasks.workunit.client.0.smithi071.stderr:49689: dumped pgs 2021-07-26T06:15:09.227 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.4 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.227 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.4 2021-07-26T06:15:09.228 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.228 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.228 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.228 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.229 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.229 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.229 INFO:tasks.workunit.client.0.smithi071.stderr:49689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2021-07-26T06:15:09.229 INFO:tasks.workunit.client.0.smithi071.stderr:49689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.4 2021-07-26T06:15:09.229 INFO:tasks.workunit.client.0.smithi071.stderr:49689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.230 INFO:tasks.workunit.client.0.smithi071.stderr:49689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.230 INFO:tasks.workunit.client.0.smithi071.stderr:49689: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2021-07-26T06:15:09.230 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.230 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.230 INFO:tasks.workunit.client.0.smithi071.stderr:49689: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.337 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1c 2021-07-26T06:15:09.338 INFO:tasks.workunit.client.0.smithi071.stderr:49854: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1c 2021-07-26T06:15:09.338 INFO:tasks.workunit.client.0.smithi071.stderr:49854: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1c 2021-07-26T06:15:09.338 INFO:tasks.workunit.client.0.smithi071.stderr:49854: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.338 INFO:tasks.workunit.client.0.smithi071.stderr:49854: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2021-07-26T06:15:09.339 INFO:tasks.workunit.client.0.smithi071.stderr:49854: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.339 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.339 INFO:tasks.workunit.client.0.smithi071.stderr:49854: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.339 INFO:tasks.workunit.client.0.smithi071.stderr:49854: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1c query 2021-07-26T06:15:09.339 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.340 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.340 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.340 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1c 2021-07-26T06:15:09.340 INFO:tasks.workunit.client.0.smithi071.stderr:49854: instructing pg 1.1c on osd.0 to scrub 2021-07-26T06:15:09.341 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.341 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2021-07-26T06:15:09.341 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.341 INFO:tasks.workunit.client.0.smithi071.stderr:49854: dumped pgs 2021-07-26T06:15:09.341 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1c 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.342 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1c 2021-07-26T06:15:09.342 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.342 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.342 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.343 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.343 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.343 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.343 INFO:tasks.workunit.client.0.smithi071.stderr:49854: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2021-07-26T06:15:09.343 INFO:tasks.workunit.client.0.smithi071.stderr:49854: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1c 2021-07-26T06:15:09.344 INFO:tasks.workunit.client.0.smithi071.stderr:49854: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.344 INFO:tasks.workunit.client.0.smithi071.stderr:49854: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2021-07-26T06:15:09.344 INFO:tasks.workunit.client.0.smithi071.stderr:49854: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.344 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.345 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.345 INFO:tasks.workunit.client.0.smithi071.stderr:49854: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.345 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.2 2021-07-26T06:15:09.346 INFO:tasks.workunit.client.0.smithi071.stderr:49677: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.2 2021-07-26T06:15:09.346 INFO:tasks.workunit.client.0.smithi071.stderr:49677: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.2 2021-07-26T06:15:09.346 INFO:tasks.workunit.client.0.smithi071.stderr:49677: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.346 INFO:tasks.workunit.client.0.smithi071.stderr:49677: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.346 INFO:tasks.workunit.client.0.smithi071.stderr:49677: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2021-07-26T06:15:09.347 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.347 INFO:tasks.workunit.client.0.smithi071.stderr:49677: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.347 INFO:tasks.workunit.client.0.smithi071.stderr:49677: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.2 query 2021-07-26T06:15:09.347 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.348 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.348 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.348 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.2 2021-07-26T06:15:09.348 INFO:tasks.workunit.client.0.smithi071.stderr:49677: instructing pg 1.2 on osd.0 to scrub 2021-07-26T06:15:09.348 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2021-07-26T06:15:09.349 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.349 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.349 INFO:tasks.workunit.client.0.smithi071.stderr:49677: dumped pgs 2021-07-26T06:15:09.349 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.2 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.349 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.2 2021-07-26T06:15:09.350 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.350 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.350 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.350 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.351 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.351 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.351 INFO:tasks.workunit.client.0.smithi071.stderr:49677: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2021-07-26T06:15:09.351 INFO:tasks.workunit.client.0.smithi071.stderr:49677: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.2 2021-07-26T06:15:09.352 INFO:tasks.workunit.client.0.smithi071.stderr:49677: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.352 INFO:tasks.workunit.client.0.smithi071.stderr:49677: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.352 INFO:tasks.workunit.client.0.smithi071.stderr:49677: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2021-07-26T06:15:09.352 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.353 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.353 INFO:tasks.workunit.client.0.smithi071.stderr:49677: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.523 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.9 2021-07-26T06:15:09.524 INFO:tasks.workunit.client.0.smithi071.stderr:49741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.9 2021-07-26T06:15:09.524 INFO:tasks.workunit.client.0.smithi071.stderr:49741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.9 2021-07-26T06:15:09.524 INFO:tasks.workunit.client.0.smithi071.stderr:49741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.525 INFO:tasks.workunit.client.0.smithi071.stderr:49741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.525 INFO:tasks.workunit.client.0.smithi071.stderr:49741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2021-07-26T06:15:09.525 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.525 INFO:tasks.workunit.client.0.smithi071.stderr:49741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.9 query 2021-07-26T06:15:09.526 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.526 INFO:tasks.workunit.client.0.smithi071.stderr:49741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.526 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.526 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.526 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.9 2021-07-26T06:15:09.527 INFO:tasks.workunit.client.0.smithi071.stderr:49741: instructing pg 1.9 on osd.1 to scrub 2021-07-26T06:15:09.527 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.527 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2021-07-26T06:15:09.527 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.528 INFO:tasks.workunit.client.0.smithi071.stderr:49741: dumped pgs 2021-07-26T06:15:09.528 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.9 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.528 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.9 2021-07-26T06:15:09.528 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.528 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.529 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.529 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.529 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.529 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.530 INFO:tasks.workunit.client.0.smithi071.stderr:49741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2021-07-26T06:15:09.530 INFO:tasks.workunit.client.0.smithi071.stderr:49741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.9 2021-07-26T06:15:09.530 INFO:tasks.workunit.client.0.smithi071.stderr:49741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.530 INFO:tasks.workunit.client.0.smithi071.stderr:49741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2021-07-26T06:15:09.530 INFO:tasks.workunit.client.0.smithi071.stderr:49741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.531 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.531 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.531 INFO:tasks.workunit.client.0.smithi071.stderr:49741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.534 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.6 2021-07-26T06:15:09.534 INFO:tasks.workunit.client.0.smithi071.stderr:49733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.6 2021-07-26T06:15:09.535 INFO:tasks.workunit.client.0.smithi071.stderr:49733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.6 2021-07-26T06:15:09.535 INFO:tasks.workunit.client.0.smithi071.stderr:49733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.535 INFO:tasks.workunit.client.0.smithi071.stderr:49733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2021-07-26T06:15:09.535 INFO:tasks.workunit.client.0.smithi071.stderr:49733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.535 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.536 INFO:tasks.workunit.client.0.smithi071.stderr:49733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.536 INFO:tasks.workunit.client.0.smithi071.stderr:49733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.6 query 2021-07-26T06:15:09.536 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.536 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.536 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.537 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.6 2021-07-26T06:15:09.537 INFO:tasks.workunit.client.0.smithi071.stderr:49733: instructing pg 1.6 on osd.6 to scrub 2021-07-26T06:15:09.537 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.537 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2021-07-26T06:15:09.537 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.538 INFO:tasks.workunit.client.0.smithi071.stderr:49733: dumped pgs 2021-07-26T06:15:09.538 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.6 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.538 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.6 2021-07-26T06:15:09.538 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.539 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.539 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.539 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.539 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.539 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.540 INFO:tasks.workunit.client.0.smithi071.stderr:49733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2021-07-26T06:15:09.540 INFO:tasks.workunit.client.0.smithi071.stderr:49733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.6 2021-07-26T06:15:09.540 INFO:tasks.workunit.client.0.smithi071.stderr:49733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.540 INFO:tasks.workunit.client.0.smithi071.stderr:49733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2021-07-26T06:15:09.540 INFO:tasks.workunit.client.0.smithi071.stderr:49733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.541 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.541 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.541 INFO:tasks.workunit.client.0.smithi071.stderr:49733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.542 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.8 2021-07-26T06:15:09.542 INFO:tasks.workunit.client.0.smithi071.stderr:49709: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.8 2021-07-26T06:15:09.542 INFO:tasks.workunit.client.0.smithi071.stderr:49709: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.8 2021-07-26T06:15:09.543 INFO:tasks.workunit.client.0.smithi071.stderr:49709: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.543 INFO:tasks.workunit.client.0.smithi071.stderr:49709: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.543 INFO:tasks.workunit.client.0.smithi071.stderr:49709: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2021-07-26T06:15:09.543 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.544 INFO:tasks.workunit.client.0.smithi071.stderr:49709: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.8 query 2021-07-26T06:15:09.544 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.544 INFO:tasks.workunit.client.0.smithi071.stderr:49709: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.544 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.544 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.545 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.8 2021-07-26T06:15:09.545 INFO:tasks.workunit.client.0.smithi071.stderr:49709: instructing pg 1.8 on osd.5 to scrub 2021-07-26T06:15:09.545 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.545 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2021-07-26T06:15:09.545 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.546 INFO:tasks.workunit.client.0.smithi071.stderr:49709: dumped pgs 2021-07-26T06:15:09.546 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.8 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.546 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.8 2021-07-26T06:15:09.546 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.547 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.547 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.547 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.547 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.548 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.548 INFO:tasks.workunit.client.0.smithi071.stderr:49709: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2021-07-26T06:15:09.549 INFO:tasks.workunit.client.0.smithi071.stderr:49709: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.8 2021-07-26T06:15:09.549 INFO:tasks.workunit.client.0.smithi071.stderr:49709: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.549 INFO:tasks.workunit.client.0.smithi071.stderr:49709: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2021-07-26T06:15:09.549 INFO:tasks.workunit.client.0.smithi071.stderr:49709: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.550 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.550 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.550 INFO:tasks.workunit.client.0.smithi071.stderr:49709: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.551 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.5 2021-07-26T06:15:09.551 INFO:tasks.workunit.client.0.smithi071.stderr:49698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.5 2021-07-26T06:15:09.552 INFO:tasks.workunit.client.0.smithi071.stderr:49698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.5 2021-07-26T06:15:09.552 INFO:tasks.workunit.client.0.smithi071.stderr:49698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.552 INFO:tasks.workunit.client.0.smithi071.stderr:49698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.552 INFO:tasks.workunit.client.0.smithi071.stderr:49698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2021-07-26T06:15:09.553 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.553 INFO:tasks.workunit.client.0.smithi071.stderr:49698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.553 INFO:tasks.workunit.client.0.smithi071.stderr:49698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.5 query 2021-07-26T06:15:09.553 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.553 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.554 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.554 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.5 2021-07-26T06:15:09.554 INFO:tasks.workunit.client.0.smithi071.stderr:49698: instructing pg 1.5 on osd.4 to scrub 2021-07-26T06:15:09.554 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2021-07-26T06:15:09.555 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.555 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.555 INFO:tasks.workunit.client.0.smithi071.stderr:49698: dumped pgs 2021-07-26T06:15:09.555 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.5 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.555 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.5 2021-07-26T06:15:09.556 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.556 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.556 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.556 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.556 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.557 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.557 INFO:tasks.workunit.client.0.smithi071.stderr:49698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2021-07-26T06:15:09.557 INFO:tasks.workunit.client.0.smithi071.stderr:49698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.5 2021-07-26T06:15:09.557 INFO:tasks.workunit.client.0.smithi071.stderr:49698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.558 INFO:tasks.workunit.client.0.smithi071.stderr:49698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.558 INFO:tasks.workunit.client.0.smithi071.stderr:49698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2021-07-26T06:15:09.558 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.558 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.558 INFO:tasks.workunit.client.0.smithi071.stderr:49698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.559 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.0 2021-07-26T06:15:09.559 INFO:tasks.workunit.client.0.smithi071.stderr:49663: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.0 2021-07-26T06:15:09.559 INFO:tasks.workunit.client.0.smithi071.stderr:49663: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:15:09.559 INFO:tasks.workunit.client.0.smithi071.stderr:49663: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.560 INFO:tasks.workunit.client.0.smithi071.stderr:49663: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.560 INFO:tasks.workunit.client.0.smithi071.stderr:49663: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:15:09.560 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.560 INFO:tasks.workunit.client.0.smithi071.stderr:49663: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.0 query 2021-07-26T06:15:09.560 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.561 INFO:tasks.workunit.client.0.smithi071.stderr:49663: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.561 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.561 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.561 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.0 2021-07-26T06:15:09.561 INFO:tasks.workunit.client.0.smithi071.stderr:49663: instructing pg 1.0 on osd.7 to scrub 2021-07-26T06:15:09.562 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.562 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2021-07-26T06:15:09.562 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.562 INFO:tasks.workunit.client.0.smithi071.stderr:49663: dumped pgs 2021-07-26T06:15:09.562 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.0 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.563 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.0 2021-07-26T06:15:09.563 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.563 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.563 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.563 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.564 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.564 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.564 INFO:tasks.workunit.client.0.smithi071.stderr:49663: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:15:09.564 INFO:tasks.workunit.client.0.smithi071.stderr:49663: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:15:09.565 INFO:tasks.workunit.client.0.smithi071.stderr:49663: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.565 INFO:tasks.workunit.client.0.smithi071.stderr:49663: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.565 INFO:tasks.workunit.client.0.smithi071.stderr:49663: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:15:09.565 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.565 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.566 INFO:tasks.workunit.client.0.smithi071.stderr:49663: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:09.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:09.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:09.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:09.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49666 2021-07-26T06:15:09.570 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.a 2021-07-26T06:15:09.570 INFO:tasks.workunit.client.0.smithi071.stderr:49718: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.a 2021-07-26T06:15:09.570 INFO:tasks.workunit.client.0.smithi071.stderr:49718: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.a 2021-07-26T06:15:09.571 INFO:tasks.workunit.client.0.smithi071.stderr:49718: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.571 INFO:tasks.workunit.client.0.smithi071.stderr:49718: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2021-07-26T06:15:09.571 INFO:tasks.workunit.client.0.smithi071.stderr:49718: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.571 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.571 INFO:tasks.workunit.client.0.smithi071.stderr:49718: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.a query 2021-07-26T06:15:09.572 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.572 INFO:tasks.workunit.client.0.smithi071.stderr:49718: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.572 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.572 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.572 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.a 2021-07-26T06:15:09.573 INFO:tasks.workunit.client.0.smithi071.stderr:49718: instructing pg 1.a on osd.1 to scrub 2021-07-26T06:15:09.573 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2021-07-26T06:15:09.573 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.573 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.574 INFO:tasks.workunit.client.0.smithi071.stderr:49718: dumped pgs 2021-07-26T06:15:09.574 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.a 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.574 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.a 2021-07-26T06:15:09.574 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.574 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.575 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.575 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.575 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.575 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.575 INFO:tasks.workunit.client.0.smithi071.stderr:49718: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2021-07-26T06:15:09.576 INFO:tasks.workunit.client.0.smithi071.stderr:49718: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.a 2021-07-26T06:15:09.576 INFO:tasks.workunit.client.0.smithi071.stderr:49718: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.576 INFO:tasks.workunit.client.0.smithi071.stderr:49718: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2021-07-26T06:15:09.576 INFO:tasks.workunit.client.0.smithi071.stderr:49718: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.577 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.577 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.577 INFO:tasks.workunit.client.0.smithi071.stderr:49718: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.609 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.3 2021-07-26T06:15:09.610 INFO:tasks.workunit.client.0.smithi071.stderr:49684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.3 2021-07-26T06:15:09.610 INFO:tasks.workunit.client.0.smithi071.stderr:49684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:15:09.610 INFO:tasks.workunit.client.0.smithi071.stderr:49684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.610 INFO:tasks.workunit.client.0.smithi071.stderr:49684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:15:09.611 INFO:tasks.workunit.client.0.smithi071.stderr:49684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.611 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.611 INFO:tasks.workunit.client.0.smithi071.stderr:49684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.3 query 2021-07-26T06:15:09.611 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.612 INFO:tasks.workunit.client.0.smithi071.stderr:49684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.612 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.612 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.612 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.3 2021-07-26T06:15:09.612 INFO:tasks.workunit.client.0.smithi071.stderr:49684: instructing pg 1.3 on osd.4 to scrub 2021-07-26T06:15:09.613 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.613 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.613 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2021-07-26T06:15:09.613 INFO:tasks.workunit.client.0.smithi071.stderr:49684: dumped pgs 2021-07-26T06:15:09.613 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.3 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.614 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.3 2021-07-26T06:15:09.614 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.614 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.614 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.615 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.615 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.615 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.615 INFO:tasks.workunit.client.0.smithi071.stderr:49684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:15:09.616 INFO:tasks.workunit.client.0.smithi071.stderr:49684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:15:09.616 INFO:tasks.workunit.client.0.smithi071.stderr:49684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.616 INFO:tasks.workunit.client.0.smithi071.stderr:49684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.616 INFO:tasks.workunit.client.0.smithi071.stderr:49684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:15:09.616 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.617 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.617 INFO:tasks.workunit.client.0.smithi071.stderr:49684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.618 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.16 2021-07-26T06:15:09.618 INFO:tasks.workunit.client.0.smithi071.stderr:49814: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.16 2021-07-26T06:15:09.618 INFO:tasks.workunit.client.0.smithi071.stderr:49814: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.16 2021-07-26T06:15:09.619 INFO:tasks.workunit.client.0.smithi071.stderr:49814: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.619 INFO:tasks.workunit.client.0.smithi071.stderr:49814: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.619 INFO:tasks.workunit.client.0.smithi071.stderr:49814: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2021-07-26T06:15:09.619 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.620 INFO:tasks.workunit.client.0.smithi071.stderr:49814: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.620 INFO:tasks.workunit.client.0.smithi071.stderr:49814: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.16 query 2021-07-26T06:15:09.620 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.620 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.620 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.621 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.16 2021-07-26T06:15:09.621 INFO:tasks.workunit.client.0.smithi071.stderr:49814: instructing pg 1.16 on osd.6 to scrub 2021-07-26T06:15:09.621 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2021-07-26T06:15:09.621 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.622 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.622 INFO:tasks.workunit.client.0.smithi071.stderr:49814: dumped pgs 2021-07-26T06:15:09.622 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.16 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.622 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.16 2021-07-26T06:15:09.622 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:49814: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2021-07-26T06:15:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:49814: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.16 2021-07-26T06:15:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:49814: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:49814: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.625 INFO:tasks.workunit.client.0.smithi071.stderr:49814: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2021-07-26T06:15:09.625 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.625 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.625 INFO:tasks.workunit.client.0.smithi071.stderr:49814: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.629 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.e 2021-07-26T06:15:09.629 INFO:tasks.workunit.client.0.smithi071.stderr:49747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.e 2021-07-26T06:15:09.630 INFO:tasks.workunit.client.0.smithi071.stderr:49747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.e 2021-07-26T06:15:09.630 INFO:tasks.workunit.client.0.smithi071.stderr:49747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.630 INFO:tasks.workunit.client.0.smithi071.stderr:49747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2021-07-26T06:15:09.630 INFO:tasks.workunit.client.0.smithi071.stderr:49747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.630 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.631 INFO:tasks.workunit.client.0.smithi071.stderr:49747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.e query 2021-07-26T06:15:09.631 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.631 INFO:tasks.workunit.client.0.smithi071.stderr:49747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.631 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.631 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.632 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.e 2021-07-26T06:15:09.632 INFO:tasks.workunit.client.0.smithi071.stderr:49747: instructing pg 1.e on osd.1 to scrub 2021-07-26T06:15:09.632 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2021-07-26T06:15:09.632 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.632 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.633 INFO:tasks.workunit.client.0.smithi071.stderr:49747: dumped pgs 2021-07-26T06:15:09.633 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.e 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.633 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.e 2021-07-26T06:15:09.633 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.633 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.634 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.634 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.634 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.634 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.635 INFO:tasks.workunit.client.0.smithi071.stderr:49747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2021-07-26T06:15:09.635 INFO:tasks.workunit.client.0.smithi071.stderr:49747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.e 2021-07-26T06:15:09.635 INFO:tasks.workunit.client.0.smithi071.stderr:49747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.635 INFO:tasks.workunit.client.0.smithi071.stderr:49747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2021-07-26T06:15:09.635 INFO:tasks.workunit.client.0.smithi071.stderr:49747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.636 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.636 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.636 INFO:tasks.workunit.client.0.smithi071.stderr:49747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.636 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1 2021-07-26T06:15:09.636 INFO:tasks.workunit.client.0.smithi071.stderr:49670: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1 2021-07-26T06:15:09.637 INFO:tasks.workunit.client.0.smithi071.stderr:49670: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1 2021-07-26T06:15:09.637 INFO:tasks.workunit.client.0.smithi071.stderr:49670: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.637 INFO:tasks.workunit.client.0.smithi071.stderr:49670: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.637 INFO:tasks.workunit.client.0.smithi071.stderr:49670: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2021-07-26T06:15:09.638 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.638 INFO:tasks.workunit.client.0.smithi071.stderr:49670: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1 query 2021-07-26T06:15:09.638 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.638 INFO:tasks.workunit.client.0.smithi071.stderr:49670: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.638 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.639 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.639 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1 2021-07-26T06:15:09.639 INFO:tasks.workunit.client.0.smithi071.stderr:49670: instructing pg 1.1 on osd.5 to scrub 2021-07-26T06:15:09.639 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.640 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.640 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2021-07-26T06:15:09.640 INFO:tasks.workunit.client.0.smithi071.stderr:49670: dumped pgs 2021-07-26T06:15:09.640 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.640 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1 2021-07-26T06:15:09.641 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.641 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.641 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.641 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.641 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.642 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.642 INFO:tasks.workunit.client.0.smithi071.stderr:49670: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2021-07-26T06:15:09.642 INFO:tasks.workunit.client.0.smithi071.stderr:49670: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1 2021-07-26T06:15:09.642 INFO:tasks.workunit.client.0.smithi071.stderr:49670: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.642 INFO:tasks.workunit.client.0.smithi071.stderr:49670: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2021-07-26T06:15:09.643 INFO:tasks.workunit.client.0.smithi071.stderr:49670: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.643 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scr/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.643 INFO:tasks.workunit.client.0.smithi071.stderr:ub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.643 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.643 INFO:tasks.workunit.client.0.smithi071.stderr:49670: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:09.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:09.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:09.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:09.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49672 2021-07-26T06:15:09.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:09.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:09.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:09.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:09.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49680 2021-07-26T06:15:09.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:09.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:09.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:09.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:09.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49685 2021-07-26T06:15:09.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:09.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:09.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:09.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:09.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49695 2021-07-26T06:15:09.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:09.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:09.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:09.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:09.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49702 2021-07-26T06:15:09.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:09.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:09.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:09.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:09.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49704 2021-07-26T06:15:09.690 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.d 2021-07-26T06:15:09.691 INFO:tasks.workunit.client.0.smithi071.stderr:49743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.d 2021-07-26T06:15:09.691 INFO:tasks.workunit.client.0.smithi071.stderr:49743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.d 2021-07-26T06:15:09.691 INFO:tasks.workunit.client.0.smithi071.stderr:49743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.691 INFO:tasks.workunit.client.0.smithi071.stderr:49743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2021-07-26T06:15:09.692 INFO:tasks.workunit.client.0.smithi071.stderr:49743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.692 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.692 INFO:tasks.workunit.client.0.smithi071.stderr:49743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.d query 2021-07-26T06:15:09.692 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.693 INFO:tasks.workunit.client.0.smithi071.stderr:49743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.693 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.693 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.693 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.d 2021-07-26T06:15:09.693 INFO:tasks.workunit.client.0.smithi071.stderr:49743: instructing pg 1.d on osd.7 to scrub 2021-07-26T06:15:09.694 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2021-07-26T06:15:09.694 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.694 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.694 INFO:tasks.workunit.client.0.smithi071.stderr:49743: dumped pgs 2021-07-26T06:15:09.694 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: recovery=true 2021-07-26T06:15:09.695 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.d 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.695 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.d 2021-07-26T06:15:09.695 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.695 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.696 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.696 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.696 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.696 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.696 INFO:tasks.workunit.client.0.smithi071.stderr:49743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2021-07-26T06:15:09.697 INFO:tasks.workunit.client.0.smithi071.stderr:49743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.d 2021-07-26T06:15:09.697 INFO:tasks.workunit.client.0.smithi071.stderr:49743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.697 INFO:tasks.workunit.client.0.smithi071.stderr:49743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.698 INFO:tasks.workunit.client.0.smithi071.stderr:49743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | se49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1e 2021-07-26T06:15:09.698 INFO:tasks.workunit.client.0.smithi071.stderr:49873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1e 2021-07-26T06:15:09.698 INFO:tasks.workunit.client.0.smithi071.stderr:49873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1e 2021-07-26T06:15:09.699 INFO:tasks.workunit.client.0.smithi071.stderr:49873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.699 INFO:tasks.workunit.client.0.smithi071.stderr:49873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2021-07-26T06:15:09.699 INFO:tasks.workunit.client.0.smithi071.stderr:49873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.699 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.700 INFO:tasks.workunit.client.0.smithi071.stderr:49873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.700 INFO:tasks.workunit.client.0.smithi071.stderr:49873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1e query 2021-07-26T06:15:09.700 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.700 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.700 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.701 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1e 2021-07-26T06:15:09.701 INFO:tasks.workunit.client.0.smithi071.stderr:49873: instructing pg 1.1e on osd.3 to scrub 2021-07-26T06:15:09.701 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.701 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.702 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2021-07-26T06:15:09.702 INFO:tasks.workunit.client.0.smithi071.stderr:49873: dumped pgs 2021-07-26T06:15:09.702 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1e 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.702 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1e 2021-07-26T06:15:09.702 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.703 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.703 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.703 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.703 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.704 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.704 INFO:tasks.workunit.client.0.smithi071.stderr:49873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2021-07-26T06:15:09.704 INFO:tasks.workunit.client.0.smithi071.stderr:49873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1e 2021-07-26T06:15:09.704 INFO:tasks.workunit.client.0.smithi071.stderr:49873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.704 INFO:tasks.workunit.client.0.smithi071.stderr:49873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.705 INFO:tasks.workunit.client.0.smithi071.stderr:49873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2021-07-26T06:15:09.705 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.705 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.705 INFO:tasks.workunit.client.0.smithi071.stderr:49873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:09.725 INFO:tasks.workunit.client.0.smithi071.stderr:et_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.725 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.726 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.726 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test true = true 2021-07-26T06:15:09.726 INFO:tasks.workunit.client.0.smithi071.stderr:49773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:199: pg_scrub_mod: return 2 2021-07-26T06:15:09.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 2 2021-07-26T06:15:09.735 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.15 2021-07-26T06:15:09.736 INFO:tasks.workunit.client.0.smithi071.stderr:49803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.15 2021-07-26T06:15:09.736 INFO:tasks.workunit.client.0.smithi071.stderr:49803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.15 2021-07-26T06:15:09.736 INFO:tasks.workunit.client.0.smithi071.stderr:49803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.737 INFO:tasks.workunit.client.0.smithi071.stderr:49803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.737 INFO:tasks.workunit.client.0.smithi071.stderr:49803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2021-07-26T06:15:09.737 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.737 INFO:tasks.workunit.client.0.smithi071.stderr:49803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:09.738 INFO:tasks.workunit.client.0.smithi071.stderr:49803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.15 query 2021-07-26T06:15:09.738 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:09.738 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:09.738 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:09.738 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.15 2021-07-26T06:15:09.739 INFO:tasks.workunit.client.0.smithi071.stderr:49803: instructing pg 1.15 on osd.1 to scrub 2021-07-26T06:15:09.739 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2021-07-26T06:15:09.739 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:09.740 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:09.740 INFO:tasks.workunit.client.0.smithi071.stderr:49803: dumped pgs 2021-07-26T06:15:09.740 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.15 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.740 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.15 2021-07-26T06:15:09.741 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:09.741 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:09.741 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:09.741 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:09.742 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:09.742 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:09.742 INFO:tasks.workunit.client.0.smithi071.stderr:49803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2021-07-26T06:15:09.742 INFO:tasks.workunit.client.0.smithi071.stderr:49803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.15 2021-07-26T06:15:09.742 INFO:tasks.workunit.client.0.smithi071.stderr:49803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:09.743 INFO:tasks.workunit.client.0.smithi071.stderr:49803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:09.743 INFO:tasks.workunit.client.0.smithi071.stderr:49803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2021-07-26T06:15:09.743 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:09.743 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:09.744 INFO:tasks.workunit.client.0.smithi071.stderr:49803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:09.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.124 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.11 2021-07-26T06:15:10.125 INFO:tasks.workunit.client.0.smithi071.stderr:49780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.11 2021-07-26T06:15:10.125 INFO:tasks.workunit.client.0.smithi071.stderr:49780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.11 2021-07-26T06:15:10.125 INFO:tasks.workunit.client.0.smithi071.stderr:49780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.125 INFO:tasks.workunit.client.0.smithi071.stderr:49780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2021-07-26T06:15:10.126 INFO:tasks.workunit.client.0.smithi071.stderr:49780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.126 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.126 INFO:tasks.workunit.client.0.smithi071.stderr:49780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.11 query 2021-07-26T06:15:10.126 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.127 INFO:tasks.workunit.client.0.smithi071.stderr:49780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.127 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.127 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.127 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.11 2021-07-26T06:15:10.127 INFO:tasks.workunit.client.0.smithi071.stderr:49780: instructing pg 1.11 on osd.0 to scrub 2021-07-26T06:15:10.128 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.128 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.128 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2021-07-26T06:15:10.128 INFO:tasks.workunit.client.0.smithi071.stderr:49780: dumped pgs 2021-07-26T06:15:10.129 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.11 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.129 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.11 2021-07-26T06:15:10.129 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.129 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.129 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.130 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.130 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.130 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.130 INFO:tasks.workunit.client.0.smithi071.stderr:49780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2021-07-26T06:15:10.131 INFO:tasks.workunit.client.0.smithi071.stderr:49780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.11 2021-07-26T06:15:10.131 INFO:tasks.workunit.client.0.smithi071.stderr:49780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.131 INFO:tasks.workunit.client.0.smithi071.stderr:49780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2021-07-26T06:15:10.131 INFO:tasks.workunit.client.0.smithi071.stderr:49780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.131 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:10.132 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.132 INFO:tasks.workunit.client.0.smithi071.stderr:49780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.163 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.12 2021-07-26T06:15:10.163 INFO:tasks.workunit.client.0.smithi071.stderr:49786: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.12 2021-07-26T06:15:10.163 INFO:tasks.workunit.client.0.smithi071.stderr:49786: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.12 2021-07-26T06:15:10.163 INFO:tasks.workunit.client.0.smithi071.stderr:49786: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.164 INFO:tasks.workunit.client.0.smithi071.stderr:49786: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2021-07-26T06:15:10.164 INFO:tasks.workunit.client.0.smithi071.stderr:49786: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.164 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.164 INFO:tasks.workunit.client.0.smithi071.stderr:49786: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.12 query 2021-07-26T06:15:10.165 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.165 INFO:tasks.workunit.client.0.smithi071.stderr:49786: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.165 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.165 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.166 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.12 2021-07-26T06:15:10.166 INFO:tasks.workunit.client.0.smithi071.stderr:49786: instructing pg 1.12 on osd.0 to scrub 2021-07-26T06:15:10.166 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.166 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.166 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2021-07-26T06:15:10.167 INFO:tasks.workunit.client.0.smithi071.stderr:49786: dumped pgs 2021-07-26T06:15:10.167 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.12 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.167 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.12 2021-07-26T06:15:10.167 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.168 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.168 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.168 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.168 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.169 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.169 INFO:tasks.workunit.client.0.smithi071.stderr:49786: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2021-07-26T06:15:10.169 INFO:tasks.workunit.client.0.smithi071.stderr:49786: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.12 2021-07-26T06:15:10.169 INFO:tasks.workunit.client.0.smithi071.stderr:49786: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.169 INFO:tasks.workunit.client.0.smithi071.stderr:49786: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2021-07-26T06:15:10.170 INFO:tasks.workunit.client.0.smithi071.stderr:49786: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.170 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:10.170 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.170 INFO:tasks.workunit.client.0.smithi071.stderr:49786: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.222 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.18 2021-07-26T06:15:10.223 INFO:tasks.workunit.client.0.smithi071.stderr:49824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.18 2021-07-26T06:15:10.223 INFO:tasks.workunit.client.0.smithi071.stderr:49824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.18 2021-07-26T06:15:10.223 INFO:tasks.workunit.client.0.smithi071.stderr:49824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.223 INFO:tasks.workunit.client.0.smithi071.stderr:49824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.223 INFO:tasks.workunit.client.0.smithi071.stderr:49824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2021-07-26T06:15:10.224 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.224 INFO:tasks.workunit.client.0.smithi071.stderr:49824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.18 query 2021-07-26T06:15:10.224 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.224 INFO:tasks.workunit.client.0.smithi071.stderr:49824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.225 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.225 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.225 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.18 2021-07-26T06:15:10.225 INFO:tasks.workunit.client.0.smithi071.stderr:49824: instructing pg 1.18 on osd.5 to scrub 2021-07-26T06:15:10.225 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.226 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2021-07-26T06:15:10.226 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.226 INFO:tasks.workunit.client.0.smithi071.stderr:49824: dumped pgs 2021-07-26T06:15:10.226 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.18 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.227 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.18 2021-07-26T06:15:10.227 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.227 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.227 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.227 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.228 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.228 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.228 INFO:tasks.workunit.client.0.smithi071.stderr:49824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2021-07-26T06:15:10.228 INFO:tasks.workunit.client.0.smithi071.stderr:49824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.18 2021-07-26T06:15:10.229 INFO:tasks.workunit.client.0.smithi071.stderr:49824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.229 INFO:tasks.workunit.client.0.smithi071.stderr:49824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.229 INFO:tasks.workunit.client.0.smithi071.stderr:49824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2021-07-26T06:15:10.229 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:15:06.993112+0000 '>' '' 2021-07-26T06:15:10.229 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.230 INFO:tasks.workunit.client.0.smithi071.stderr:49824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.230 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.7 2021-07-26T06:15:10.231 INFO:tasks.workunit.client.0.smithi071.stderr:49736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.7 2021-07-26T06:15:10.231 INFO:tasks.workunit.client.0.smithi071.stderr:49736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.7 2021-07-26T06:15:10.231 INFO:tasks.workunit.client.0.smithi071.stderr:49736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.231 INFO:tasks.workunit.client.0.smithi071.stderr:49736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2021-07-26T06:15:10.232 INFO:tasks.workunit.client.0.smithi071.stderr:49736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.232 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.232 INFO:tasks.workunit.client.0.smithi071.stderr:49736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.7 query 2021-07-26T06:15:10.232 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.232 INFO:tasks.workunit.client.0.smithi071.stderr:49736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.233 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.233 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.233 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.7 2021-07-26T06:15:10.233 INFO:tasks.workunit.client.0.smithi071.stderr:49736: instructing pg 1.7 on osd.6 to scrub 2021-07-26T06:15:10.233 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2021-07-26T06:15:10.234 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.234 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.234 INFO:tasks.workunit.client.0.smithi071.stderr:49736: dumped pgs 2021-07-26T06:15:10.234 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.7 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.234 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.7 2021-07-26T06:15:10.235 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.235 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.235 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.235 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.236 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.236 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.236 INFO:tasks.workunit.client.0.smithi071.stderr:49736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2021-07-26T06:15:10.236 INFO:tasks.workunit.client.0.smithi071.stderr:49736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.7 2021-07-26T06:15:10.236 INFO:tasks.workunit.client.0.smithi071.stderr:49736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.237 INFO:tasks.workunit.client.0.smithi071.stderr:49736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.237 INFO:tasks.workunit.client.0.smithi071.stderr:49736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2021-07-26T06:15:10.237 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:10.237 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.238 INFO:tasks.workunit.client.0.smithi071.stderr:49736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49706 2021-07-26T06:15:10.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49710 2021-07-26T06:15:10.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49715 2021-07-26T06:15:10.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49719 2021-07-26T06:15:10.278 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.f 2021-07-26T06:15:10.279 INFO:tasks.workunit.client.0.smithi071.stderr:49753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.f 2021-07-26T06:15:10.279 INFO:tasks.workunit.client.0.smithi071.stderr:49753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.f 2021-07-26T06:15:10.279 INFO:tasks.workunit.client.0.smithi071.stderr:49753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.279 INFO:tasks.workunit.client.0.smithi071.stderr:49753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.280 INFO:tasks.workunit.client.0.smithi071.stderr:49753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2021-07-26T06:15:10.280 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.280 INFO:tasks.workunit.client.0.smithi071.stderr:49753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.f query 2021-07-26T06:15:10.280 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.281 INFO:tasks.workunit.client.0.smithi071.stderr:49753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.281 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.281 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.281 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.f 2021-07-26T06:15:10.281 INFO:tasks.workunit.client.0.smithi071.stderr:49753: instructing pg 1.f on osd.6 to scrub 2021-07-26T06:15:10.282 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.282 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2021-07-26T06:15:10.282 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.282 INFO:tasks.workunit.client.0.smithi071.stderr:49753: dumped pgs 2021-07-26T06:15:10.283 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.f 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.283 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.f 2021-07-26T06:15:10.283 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.283 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.283 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.284 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.284 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.284 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.284 INFO:tasks.workunit.client.0.smithi071.stderr:49753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2021-07-26T06:15:10.285 INFO:tasks.workunit.client.0.smithi071.stderr:49753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.f 2021-07-26T06:15:10.285 INFO:tasks.workunit.client.0.smithi071.stderr:49753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.285 INFO:tasks.workunit.client.0.smithi071.stderr:49753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2021-07-26T06:15:10.285 INFO:tasks.workunit.client.0.smithi071.stderr:49753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.285 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:10.286 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.286 INFO:tasks.workunit.client.0.smithi071.stderr:49753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.296 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1d 2021-07-26T06:15:10.297 INFO:tasks.workunit.client.0.smithi071.stderr:49863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1d 2021-07-26T06:15:10.297 INFO:tasks.workunit.client.0.smithi071.stderr:49863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1d 2021-07-26T06:15:10.297 INFO:tasks.workunit.client.0.smithi071.stderr:49863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.297 INFO:tasks.workunit.client.0.smithi071.stderr:49863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2021-07-26T06:15:10.298 INFO:tasks.workunit.client.0.smithi071.stderr:49863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.298 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.298 INFO:tasks.workunit.client.0.smithi071.stderr:49863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1d query 2021-07-26T06:15:10.298 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.299 INFO:tasks.workunit.client.0.smithi071.stderr:49863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.299 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.299 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.299 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1d 2021-07-26T06:15:10.300 INFO:tasks.workunit.client.0.smithi071.stderr:49863: instructing pg 1.1d on osd.1 to scrub 2021-07-26T06:15:10.300 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2021-07-26T06:15:10.300 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.300 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.301 INFO:tasks.workunit.client.0.smithi071.stderr:49863: dumped pgs 2021-07-26T06:15:10.301 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1d 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.301 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1d 2021-07-26T06:15:10.301 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.301 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.302 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.302 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.302 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.302 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.302 INFO:tasks.workunit.client.0.smithi071.stderr:49863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2021-07-26T06:15:10.303 INFO:tasks.workunit.client.0.smithi071.stderr:49863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1d 2021-07-26T06:15:10.303 INFO:tasks.workunit.client.0.smithi071.stderr:49863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.303 INFO:tasks.workunit.client.0.smithi071.stderr:49863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.303 INFO:tasks.workunit.client.0.smithi071.stderr:49863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2021-07-26T06:15:10.304 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:10.304 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.304 INFO:tasks.workunit.client.0.smithi071.stderr:49863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.305 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1b 2021-07-26T06:15:10.305 INFO:tasks.workunit.client.0.smithi071.stderr:49851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1b 2021-07-26T06:15:10.306 INFO:tasks.workunit.client.0.smithi071.stderr:49851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1b 2021-07-26T06:15:10.306 INFO:tasks.workunit.client.0.smithi071.stderr:49851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.306 INFO:tasks.workunit.client.0.smithi071.stderr:49851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2021-07-26T06:15:10.306 INFO:tasks.workunit.client.0.smithi071.stderr:49851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.307 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.307 INFO:tasks.workunit.client.0.smithi071.stderr:49851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.307 INFO:tasks.workunit.client.0.smithi071.stderr:49851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1b query 2021-07-26T06:15:10.307 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.308 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.308 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.308 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1b 2021-07-26T06:15:10.308 INFO:tasks.workunit.client.0.smithi071.stderr:49851: instructing pg 1.1b on osd.0 to scrub 2021-07-26T06:15:10.308 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2021-07-26T06:15:10.309 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.309 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.309 INFO:tasks.workunit.client.0.smithi071.stderr:49851: dumped pgs 2021-07-26T06:15:10.309 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1b 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.310 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1b 2021-07-26T06:15:10.310 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.310 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.310 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.311 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.311 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.311 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.311 INFO:tasks.workunit.client.0.smithi071.stderr:49851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2021-07-26T06:15:10.311 INFO:tasks.workunit.client.0.smithi071.stderr:49851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1b 2021-07-26T06:15:10.312 INFO:tasks.workunit.client.0.smithi071.stderr:49851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.312 INFO:tasks.workunit.client.0.smithi071.stderr:49851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.312 INFO:tasks.workunit.client.0.smithi071.stderr:49851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2021-07-26T06:15:10.312 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:10.313 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.313 INFO:tasks.workunit.client.0.smithi071.stderr:49851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.314 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1a 2021-07-26T06:15:10.314 INFO:tasks.workunit.client.0.smithi071.stderr:49841: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1a 2021-07-26T06:15:10.314 INFO:tasks.workunit.client.0.smithi071.stderr:49841: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1a 2021-07-26T06:15:10.315 INFO:tasks.workunit.client.0.smithi071.stderr:49841: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.315 INFO:tasks.workunit.client.0.smithi071.stderr:49841: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.315 INFO:tasks.workunit.client.0.smithi071.stderr:49841: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2021-07-26T06:15:10.315 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.316 INFO:tasks.workunit.client.0.smithi071.stderr:49841: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.316 INFO:tasks.workunit.client.0.smithi071.stderr:49841: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1a query 2021-07-26T06:15:10.316 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.316 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.317 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.317 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1a 2021-07-26T06:15:10.317 INFO:tasks.workunit.client.0.smithi071.stderr:49841: instructing pg 1.1a on osd.1 to scrub 2021-07-26T06:15:10.318 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2021-07-26T06:15:10.318 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.318 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.318 INFO:tasks.workunit.client.0.smithi071.stderr:49841: dumped pgs 2021-07-26T06:15:10.318 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1a 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.319 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1a 2021-07-26T06:15:10.319 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.319 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.319 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.320 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.320 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.320 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.320 INFO:tasks.workunit.client.0.smithi071.stderr:49841: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2021-07-26T06:15:10.321 INFO:tasks.workunit.client.0.smithi071.stderr:49841: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1a 2021-07-26T06:15:10.321 INFO:tasks.workunit.client.0.smithi071.stderr:49841: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.321 INFO:tasks.workunit.client.0.smithi071.stderr:49841: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.321 INFO:tasks.workunit.client.0.smithi071.stderr:49841: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2021-07-26T06:15:10.321 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:10.322 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.322 INFO:tasks.workunit.client.0.smithi071.stderr:49841: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.327 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.19 2021-07-26T06:15:10.328 INFO:tasks.workunit.client.0.smithi071.stderr:49836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.19 2021-07-26T06:15:10.328 INFO:tasks.workunit.client.0.smithi071.stderr:49836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.19 2021-07-26T06:15:10.328 INFO:tasks.workunit.client.0.smithi071.stderr:49836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.328 INFO:tasks.workunit.client.0.smithi071.stderr:49836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2021-07-26T06:15:10.329 INFO:tasks.workunit.client.0.smithi071.stderr:49836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.329 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.329 INFO:tasks.workunit.client.0.smithi071.stderr:49836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.19 query 2021-07-26T06:15:10.329 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.330 INFO:tasks.workunit.client.0.smithi071.stderr:49836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.330 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.330 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.330 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.19 2021-07-26T06:15:10.330 INFO:tasks.workunit.client.0.smithi071.stderr:49836: instructing pg 1.19 on osd.3 to scrub 2021-07-26T06:15:10.331 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.331 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.331 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2021-07-26T06:15:10.331 INFO:tasks.workunit.client.0.smithi071.stderr:49836: dumped pgs 2021-07-26T06:15:10.331 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.19 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.332 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.19 2021-07-26T06:15:10.332 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.332 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.332 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.333 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.333 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.333 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.333 INFO:tasks.workunit.client.0.smithi071.stderr:49836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2021-07-26T06:15:10.333 INFO:tasks.workunit.client.0.smithi071.stderr:49836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.19 2021-07-26T06:15:10.334 INFO:tasks.workunit.client.0.smithi071.stderr:49836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.334 INFO:tasks.workunit.client.0.smithi071.stderr:49836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2021-07-26T06:15:10.334 INFO:tasks.workunit.client.0.smithi071.stderr:49836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.334 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:10.335 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.335 INFO:tasks.workunit.client.0.smithi071.stderr:49836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.336 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.13 2021-07-26T06:15:10.337 INFO:tasks.workunit.client.0.smithi071.stderr:49790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.13 2021-07-26T06:15:10.337 INFO:tasks.workunit.client.0.smithi071.stderr:49790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.13 2021-07-26T06:15:10.337 INFO:tasks.workunit.client.0.smithi071.stderr:49790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.337 INFO:tasks.workunit.client.0.smithi071.stderr:49790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.338 INFO:tasks.workunit.client.0.smithi071.stderr:49790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2021-07-26T06:15:10.338 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.338 INFO:tasks.workunit.client.0.smithi071.stderr:49790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.338 INFO:tasks.workunit.client.0.smithi071.stderr:49790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.13 query 2021-07-26T06:15:10.339 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.339 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.339 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.339 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.13 2021-07-26T06:15:10.340 INFO:tasks.workunit.client.0.smithi071.stderr:49790: instructing pg 1.13 on osd.4 to scrub 2021-07-26T06:15:10.340 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2021-07-26T06:15:10.340 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.340 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.340 INFO:tasks.workunit.client.0.smithi071.stderr:49790: dumped pgs 2021-07-26T06:15:10.341 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.13 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.341 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.13 2021-07-26T06:15:10.341 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.341 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.342 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.342 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.342 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.342 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.342 INFO:tasks.workunit.client.0.smithi071.stderr:49790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2021-07-26T06:15:10.343 INFO:tasks.workunit.client.0.smithi071.stderr:49790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.13 2021-07-26T06:15:10.343 INFO:tasks.workunit.client.0.smithi071.stderr:49790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.343 INFO:tasks.workunit.client.0.smithi071.stderr:49790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.343 INFO:tasks.workunit.client.0.smithi071.stderr:49790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2021-07-26T06:15:10.344 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:10.344 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.344 INFO:tasks.workunit.client.0.smithi071.stderr:49790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.345 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1f 2021-07-26T06:15:10.345 INFO:tasks.workunit.client.0.smithi071.stderr:49879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1f 2021-07-26T06:15:10.345 INFO:tasks.workunit.client.0.smithi071.stderr:49879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1f 2021-07-26T06:15:10.346 INFO:tasks.workunit.client.0.smithi071.stderr:49879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.346 INFO:tasks.workunit.client.0.smithi071.stderr:49879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.346 INFO:tasks.workunit.client.0.smithi071.stderr:49879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2021-07-26T06:15:10.346 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.346 INFO:tasks.workunit.client.0.smithi071.stderr:49879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1f query 2021-07-26T06:15:10.347 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.347 INFO:tasks.workunit.client.0.smithi071.stderr:49879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.347 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.347 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.348 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1f 2021-07-26T06:15:10.348 INFO:tasks.workunit.client.0.smithi071.stderr:49879: instructing pg 1.1f on osd.5 to scrub 2021-07-26T06:15:10.348 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2021-07-26T06:15:10.348 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.348 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.349 INFO:tasks.workunit.client.0.smithi071.stderr:49879: dumped pgs 2021-07-26T06:15:10.349 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1f 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.349 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1f 2021-07-26T06:15:10.349 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.349 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.350 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.350 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.350 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.350 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.351 INFO:tasks.workunit.client.0.smithi071.stderr:49879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2021-07-26T06:15:10.351 INFO:tasks.workunit.client.0.smithi071.stderr:49879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1f 2021-07-26T06:15:10.351 INFO:tasks.workunit.client.0.smithi071.stderr:49879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.351 INFO:tasks.workunit.client.0.smithi071.stderr:49879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2021-07-26T06:15:10.351 INFO:tasks.workunit.client.0.smithi071.stderr:49879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.352 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:10.352 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.352 INFO:tasks.workunit.client.0.smithi071.stderr:49879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.353 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.b 2021-07-26T06:15:10.354 INFO:tasks.workunit.client.0.smithi071.stderr:49723: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.b 2021-07-26T06:15:10.354 INFO:tasks.workunit.client.0.smithi071.stderr:49723: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.b 2021-07-26T06:15:10.354 INFO:tasks.workunit.client.0.smithi071.stderr:49723: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.354 INFO:tasks.workunit.client.0.smithi071.stderr:49723: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.355 INFO:tasks.workunit.client.0.smithi071.stderr:49723: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2021-07-26T06:15:10.355 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.355 INFO:tasks.workunit.client.0.smithi071.stderr:49723: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.b query 2021-07-26T06:15:10.355 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.356 INFO:tasks.workunit.client.0.smithi071.stderr:49723: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.356 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.356 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.356 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.b 2021-07-26T06:15:10.356 INFO:tasks.workunit.client.0.smithi071.stderr:49723: instructing pg 1.b on osd.1 to scrub 2021-07-26T06:15:10.357 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2021-07-26T06:15:10.357 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.357 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.357 INFO:tasks.workunit.client.0.smithi071.stderr:49723: dumped pgs 2021-07-26T06:15:10.358 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.b 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.358 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.b 2021-07-26T06:15:10.358 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.358 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.358 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.359 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.359 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.359 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.359 INFO:tasks.workunit.client.0.smithi071.stderr:49723: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2021-07-26T06:15:10.359 INFO:tasks.workunit.client.0.smithi071.stderr:49723: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.b 2021-07-26T06:15:10.360 INFO:tasks.workunit.client.0.smithi071.stderr:49723: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.360 INFO:tasks.workunit.client.0.smithi071.stderr:49723: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.360 INFO:tasks.workunit.client.0.smithi071.stderr:49723: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2021-07-26T06:15:10.360 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:10.361 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.361 INFO:tasks.workunit.client.0.smithi071.stderr:49723: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49722 2021-07-26T06:15:10.580 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.14 2021-07-26T06:15:10.580 INFO:tasks.workunit.client.0.smithi071.stderr:49799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.14 2021-07-26T06:15:10.580 INFO:tasks.workunit.client.0.smithi071.stderr:49799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.14 2021-07-26T06:15:10.580 INFO:tasks.workunit.client.0.smithi071.stderr:49799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.581 INFO:tasks.workunit.client.0.smithi071.stderr:49799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2021-07-26T06:15:10.581 INFO:tasks.workunit.client.0.smithi071.stderr:49799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.581 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.581 INFO:tasks.workunit.client.0.smithi071.stderr:49799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.582 INFO:tasks.workunit.client.0.smithi071.stderr:49799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.14 query 2021-07-26T06:15:10.582 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.582 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.582 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.582 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.14 2021-07-26T06:15:10.583 INFO:tasks.workunit.client.0.smithi071.stderr:49799: instructing pg 1.14 on osd.2 to scrub 2021-07-26T06:15:10.583 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2021-07-26T06:15:10.583 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.583 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.584 INFO:tasks.workunit.client.0.smithi071.stderr:49799: dumped pgs 2021-07-26T06:15:10.584 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.14 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.584 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.14 2021-07-26T06:15:10.584 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.584 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.585 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.585 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.585 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.585 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.586 INFO:tasks.workunit.client.0.smithi071.stderr:49799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2021-07-26T06:15:10.586 INFO:tasks.workunit.client.0.smithi071.stderr:49799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.14 2021-07-26T06:15:10.586 INFO:tasks.workunit.client.0.smithi071.stderr:49799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.586 INFO:tasks.workunit.client.0.smithi071.stderr:49799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.587 INFO:tasks.workunit.client.0.smithi071.stderr:49799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2021-07-26T06:15:10.587 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:15:08.281322+0000 '>' '' 2021-07-26T06:15:10.587 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.587 INFO:tasks.workunit.client.0.smithi071.stderr:49799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.600 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.c 2021-07-26T06:15:10.600 INFO:tasks.workunit.client.0.smithi071.stderr:49746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.c 2021-07-26T06:15:10.600 INFO:tasks.workunit.client.0.smithi071.stderr:49746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.c 2021-07-26T06:15:10.601 INFO:tasks.workunit.client.0.smithi071.stderr:49746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.601 INFO:tasks.workunit.client.0.smithi071.stderr:49746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.601 INFO:tasks.workunit.client.0.smithi071.stderr:49746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2021-07-26T06:15:10.601 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.601 INFO:tasks.workunit.client.0.smithi071.stderr:49746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.c query 2021-07-26T06:15:10.602 INFO:tasks.workunit.client.0.smithi071.stderr:49746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2021-07-26T06:15:10.602 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2021-07-26T06:15:10.602 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2021-07-26T06:15:10.602 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2021-07-26T06:15:10.603 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.c 2021-07-26T06:15:10.603 INFO:tasks.workunit.client.0.smithi071.stderr:49746: instructing pg 1.c on osd.3 to scrub 2021-07-26T06:15:10.603 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2021-07-26T06:15:10.603 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2021-07-26T06:15:10.603 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2021-07-26T06:15:10.604 INFO:tasks.workunit.client.0.smithi071.stderr:49746: dumped pgs 2021-07-26T06:15:10.604 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.c 2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.604 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.c 2021-07-26T06:15:10.604 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2021-07-26T06:13:22.659593+0000 2021-07-26T06:15:10.605 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2021-07-26T06:15:10.605 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2021-07-26T06:15:10.605 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2021-07-26T06:15:10.605 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2021-07-26T06:15:10.605 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2021-07-26T06:15:10.606 INFO:tasks.workunit.client.0.smithi071.stderr:49746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2021-07-26T06:15:10.606 INFO:tasks.workunit.client.0.smithi071.stderr:49746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.c 2021-07-26T06:15:10.606 INFO:tasks.workunit.client.0.smithi071.stderr:49746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:15:10.606 INFO:tasks.workunit.client.0.smithi071.stderr:49746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:15:10.607 INFO:tasks.workunit.client.0.smithi071.stderr:49746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2021-07-26T06:15:10.607 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:10.607 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.607 INFO:tasks.workunit.client.0.smithi071.stderr:49746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2021-07-26T06:15:10.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:15:10.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49726 2021-07-26T06:15:10.609 INFO:tasks.workunit.client.0.smithi071.stderr:lect(.pgid=="1.d") | .last_scrub_stamp' 2021-07-26T06:15:10.609 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2021-07-26T06:13:22.659593+0000 '>' '' 2021-07-26T06:15:10.609 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2021-07-26T06:15:10.609 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test true = true 2021-07-26T06:15:10.610 INFO:tasks.workunit.client.0.smithi071.stderr:49743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:199: pg_scrub_mod: return 2 2021-07-26T06:15:10.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 2 2021-07-26T06:15:10.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=2 2021-07-26T06:15:10.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 2 -eq 2 2021-07-26T06:15:10.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: expr 0 + 1 2021-07-26T06:15:10.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: recov_scrub_count=1 2021-07-26T06:15:10.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49730 2021-07-26T06:15:10.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49749 2021-07-26T06:15:10.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49763 2021-07-26T06:15:10.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=2 2021-07-26T06:15:10.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 2 -eq 2 2021-07-26T06:15:10.614 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: expr 1 + 1 2021-07-26T06:15:10.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: recov_scrub_count=2 2021-07-26T06:15:10.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49770 2021-07-26T06:15:10.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49781 2021-07-26T06:15:10.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49783 2021-07-26T06:15:10.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49791 2021-07-26T06:15:10.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49795 2021-07-26T06:15:10.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49800 2021-07-26T06:15:10.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49811 2021-07-26T06:15:10.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49816 2021-07-26T06:15:10.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49826 2021-07-26T06:15:10.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49837 2021-07-26T06:15:10.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49845 2021-07-26T06:15:10.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49848 2021-07-26T06:15:10.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49856 2021-07-26T06:15:10.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49868 2021-07-26T06:15:10.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2021-07-26T06:15:10.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 49876 2021-07-26T06:15:10.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2021-07-26T06:15:10.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2021-07-26T06:15:10.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2021-07-26T06:15:10.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:223: wait_background_check: eval 'pids='\'''\''' 2021-07-26T06:15:10.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:223: wait_background_check: pids= 2021-07-26T06:15:10.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:225: wait_background_check: return 0 2021-07-26T06:15:10.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: return_code=0 2021-07-26T06:15:10.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2021-07-26T06:15:10.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:293: TEST_recovery_scrub_2: ERRORS=0 2021-07-26T06:15:10.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:294: TEST_recovery_scrub_2: test 2 -eq 0 2021-07-26T06:15:10.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2021-07-26T06:15:10.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2021-07-26T06:15:10.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/mon.a.pid 2021-07-26T06:15:10.633 INFO:tasks.workunit.client.0.smithi071.stderr:td/osd-recovery-scrub/osd.5.pid 2021-07-26T06:15:10.633 INFO:tasks.workunit.client.0.smithi071.stderr:td/osd-recovery-scrub/osd.4.pid 2021-07-26T06:15:10.633 INFO:tasks.workunit.client.0.smithi071.stderr:td/osd-recovery-scrub/osd.0.pid 2021-07-26T06:15:10.633 INFO:tasks.workunit.client.0.smithi071.stderr:td/osd-recovery-scrub/mgr.x.pid 2021-07-26T06:15:10.634 INFO:tasks.workunit.client.0.smithi071.stderr:td/osd-recovery-scrub/osd.3.pid 2021-07-26T06:15:10.634 INFO:tasks.workunit.client.0.smithi071.stderr:td/osd-recovery-scrub/osd.6.pid 2021-07-26T06:15:10.634 INFO:tasks.workunit.client.0.smithi071.stderr:td/osd-recovery-scrub/osd.1.pid 2021-07-26T06:15:10.634 INFO:tasks.workunit.client.0.smithi071.stderr:td/osd-recovery-scrub/osd.7.pid 2021-07-26T06:15:10.634 INFO:tasks.workunit.client.0.smithi071.stderr:td/osd-recovery-scrub/osd.2.pid' 2021-07-26T06:15:10.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/osd.6.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.2.pid 2021-07-26T06:15:10.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: pid='36552 2021-07-26T06:15:10.635 INFO:tasks.workunit.client.0.smithi071.stderr:44622 2021-07-26T06:15:10.635 INFO:tasks.workunit.client.0.smithi071.stderr:43223 2021-07-26T06:15:10.636 INFO:tasks.workunit.client.0.smithi071.stderr:37619 2021-07-26T06:15:10.636 INFO:tasks.workunit.client.0.smithi071.stderr:36639 2021-07-26T06:15:10.636 INFO:tasks.workunit.client.0.smithi071.stderr:41824 2021-07-26T06:15:10.636 INFO:tasks.workunit.client.0.smithi071.stderr:46024 2021-07-26T06:15:10.636 INFO:tasks.workunit.client.0.smithi071.stderr:39018 2021-07-26T06:15:10.637 INFO:tasks.workunit.client.0.smithi071.stderr:47431 2021-07-26T06:15:10.637 INFO:tasks.workunit.client.0.smithi071.stderr:40425' 2021-07-26T06:15:10.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: kill -0 36552 44622 43223 37619 36639 41824 46024 39018 47431 40425 2021-07-26T06:15:10.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2021-07-26T06:15:10.638 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:15:10.638 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:15:10.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:15:10.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:15:10.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:15:11.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:15:11.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:312: TEST_recovery_scrub_2: declare -a err_strings 2021-07-26T06:15:11.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:313: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2021-07-26T06:15:11.900 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: expr 8 - 1 2021-07-26T06:15:11.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: seq 0 7 2021-07-26T06:15:11.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2021-07-26T06:15:11.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2021-07-26T06:15:11.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2021-07-26T06:15:11.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2021-07-26T06:15:11.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2021-07-26T06:15:11.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2021-07-26T06:15:11.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2021-07-26T06:15:11.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2021-07-26T06:15:11.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2021-07-26T06:15:11.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:321: TEST_recovery_scrub_2: found=false 2021-07-26T06:15:11.929 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: expr 8 - 1 2021-07-26T06:15:11.929 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: seq 0 7 2021-07-26T06:15:11.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2021-07-26T06:15:11.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2021-07-26T06:15:11.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2021-07-26T06:15:11.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2021-07-26T06:15:11.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2021-07-26T06:15:11.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2021-07-26T06:15:11.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2021-07-26T06:15:11.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:11.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2021-07-26T06:15:11.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: '[' false = true ']' 2021-07-26T06:15:11.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:335: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2021-07-26T06:15:11.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2021-07-26T06:15:11.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:15:11.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2021-07-26T06:15:11.954 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:15:11.955 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:15:11.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:15:11.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:15:11.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:15:11.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:15:11.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:15:11.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:15:11.963 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:15:11.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:15:11.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:15:11.965 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:15:11.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:15:11.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:15:11.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:15:11.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:15:11.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:15:11.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:15:11.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2021-07-26T06:15:12.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:15:12.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:12.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:15:12.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27826 2021-07-26T06:15:12.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:15:12.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:15:12.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:337: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2021-07-26T06:15:12.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:343: TEST_recovery_scrub_2: echo 'TEST PASSED' 2021-07-26T06:15:12.230 INFO:tasks.workunit.client.0.smithi071.stdout:TEST PASSED 2021-07-26T06:15:12.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:344: TEST_recovery_scrub_2: return 0 2021-07-26T06:15:12.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:15:12.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-recovery-scrub 0 2021-07-26T06:15:12.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2021-07-26T06:15:12.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:15:12.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2021-07-26T06:15:12.233 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:15:12.233 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:15:12.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:15:12.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:15:12.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:15:12.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:15:12.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:15:12.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:15:12.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:15:12.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:15:12.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:15:12.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:15:12.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:15:12.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:15:12.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:15:12.239 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:15:12.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:15:12.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:15:12.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2021-07-26T06:15:12.239 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:15:12.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:12.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27826 2021-07-26T06:15:12.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27826 2021-07-26T06:15:12.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:15:12.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:15:12.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:15:12.241 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:15:12.242 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:15:12.311 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2021-07-26T06:15:12.312 DEBUG:teuthology.orchestra.run.smithi071:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2021-07-26T06:15:12.372 INFO:tasks.workunit.client.0.smithi071.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:15:12.380 INFO:tasks.workunit.client.0.smithi071.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:15:12.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-07-26T06:15:12.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-07-26T06:15:12.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:15:12.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:15:12.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:15:12.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:15:12.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:15:12.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:15:12.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-scrub-dump 2021-07-26T06:15:12.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:25: run: local dir=td/osd-scrub-dump 2021-07-26T06:15:12.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:26: run: shift 2021-07-26T06:15:12.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:27: run: local SLEEP=0 2021-07-26T06:15:12.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:28: run: local CHUNK_MAX=5 2021-07-26T06:15:12.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: export CEPH_MON=127.0.0.1:7184 2021-07-26T06:15:12.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: CEPH_MON=127.0.0.1:7184 2021-07-26T06:15:12.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: export CEPH_ARGS 2021-07-26T06:15:12.384 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:32: run: uuidgen 2021-07-26T06:15:12.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:32: run: CEPH_ARGS+='--fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none ' 2021-07-26T06:15:12.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:33: run: CEPH_ARGS+='--mon-host=127.0.0.1:7184 ' 2021-07-26T06:15:12.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:34: run: CEPH_ARGS+='--osd_max_scrubs=4 ' 2021-07-26T06:15:12.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:35: run: CEPH_ARGS+='--osd_scrub_sleep=0 ' 2021-07-26T06:15:12.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:36: run: CEPH_ARGS+='--osd_scrub_chunk_max=5 ' 2021-07-26T06:15:12.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:37: run: CEPH_ARGS+='--osd_scrub_sleep=2 ' 2021-07-26T06:15:12.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:38: run: CEPH_ARGS+='--osd_pool_default_size=3 ' 2021-07-26T06:15:12.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:40: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:15:12.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:41: run: set 2021-07-26T06:15:12.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:41: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:15:12.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:41: run: local funcs=TEST_recover_unexpected 2021-07-26T06:15:12.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:42: run: for func in $funcs 2021-07-26T06:15:12.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:43: run: setup td/osd-scrub-dump 2021-07-26T06:15:12.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-dump 2021-07-26T06:15:12.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-dump 2021-07-26T06:15:12.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2021-07-26T06:15:12.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:15:12.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2021-07-26T06:15:12.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:15:12.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:15:12.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:15:12.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:15:12.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:15:12.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:15:12.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:15:12.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:15:12.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:15:12.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:15:12.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:15:12.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:15:12.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:15:12.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:15:12.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:15:12.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:15:12.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:15:12.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:15:12.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2021-07-26T06:15:12.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:15:12.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:12.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:15:12.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53231 2021-07-26T06:15:12.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:15:12.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:15:12.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-dump 2021-07-26T06:15:12.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:15:12.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:12.400 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:15:12.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53231 2021-07-26T06:15:12.400 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:15:12.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:15:12.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:15:12.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:15:12.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-dump 1' TERM HUP INT 2021-07-26T06:15:12.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:44: run: TEST_recover_unexpected td/osd-scrub-dump 2021-07-26T06:15:12.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:50: TEST_recover_unexpected: local dir=td/osd-scrub-dump 2021-07-26T06:15:12.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:51: TEST_recover_unexpected: shift 2021-07-26T06:15:12.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:52: TEST_recover_unexpected: local OSDS=6 2021-07-26T06:15:12.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:53: TEST_recover_unexpected: local PGS=16 2021-07-26T06:15:12.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:54: TEST_recover_unexpected: local POOLS=3 2021-07-26T06:15:12.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:55: TEST_recover_unexpected: local OBJS=1000 2021-07-26T06:15:12.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:57: TEST_recover_unexpected: run_mon td/osd-scrub-dump a 2021-07-26T06:15:12.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-dump 2021-07-26T06:15:12.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:15:12.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:15:12.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:15:12.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-dump/a 2021-07-26T06:15:12.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-dump/a --run-dir=td/osd-scrub-dump 2021-07-26T06:15:12.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:15:12.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:15:12.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:15:12.771 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:15:12.772 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:12.772 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:15:12.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:15:12.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-dump/a '--log-file=td/osd-scrub-dump/$name.log' '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-dump/log --run-dir=td/osd-scrub-dump '--pid-file=td/osd-scrub-dump/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:15:13.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:15:13.049 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:15:13.049 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:15:13.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:15:13.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:15:13.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:15:13.050 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:15:13.051 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:15:13.051 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:15:13.051 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:15:13.051 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:13.052 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:15:13.052 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-mon.a.asok 2021-07-26T06:15:13.052 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:15:13.052 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.53231/ceph-mon.a.asok config get fsid 2021-07-26T06:15:13.145 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:15:13.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:15:13.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:15:13.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:15:13.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:15:13.147 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:15:13.147 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:15:13.147 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:15:13.147 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:15:13.148 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:13.148 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:15:13.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-mon.a.asok 2021-07-26T06:15:13.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:15:13.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.53231/ceph-mon.a.asok config get mon_host 2021-07-26T06:15:13.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:58: TEST_recover_unexpected: run_mgr td/osd-scrub-dump x 2021-07-26T06:15:13.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-dump 2021-07-26T06:15:13.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:15:13.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:15:13.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:15:13.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-dump/x 2021-07-26T06:15:13.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:15:14.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:15:14.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:15:14.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:15:14.109 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:15:14.109 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:14.110 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:15:14.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:15:14.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:15:14.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-dump/x '--log-file=td/osd-scrub-dump/$name.log' '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --run-dir=td/osd-scrub-dump '--pid-file=td/osd-scrub-dump/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:15:14.138 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:15:14.138 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: seq 0 5 2021-07-26T06:15:14.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:14.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: run_osd td/osd-scrub-dump 0 2021-07-26T06:15:14.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2021-07-26T06:15:14.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:15:14.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:15:14.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:15:14.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/0 2021-07-26T06:15:14.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 ' 2021-07-26T06:15:14.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:15:14.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:15:14.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:15:14.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/0' 2021-07-26T06:15:14.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/0/journal' 2021-07-26T06:15:14.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:15:14.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:15:14.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:15:14.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2021-07-26T06:15:14.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:15:14.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:15:14.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:15:14.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:15:14.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:14.149 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:15:14.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:15:14.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:15:14.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:15:14.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:15:14.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:15:14.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2021-07-26T06:15:14.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2021-07-26T06:15:14.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:15:14.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:15:14.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:15:14.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:15:14.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:15:14.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/0 2021-07-26T06:15:14.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:15:14.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4bb3c2bc-fb41-42d9-812e-19e2ddf04896 2021-07-26T06:15:14.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4bb3c2bc-fb41-42d9-812e-19e2ddf04896' 2021-07-26T06:15:14.153 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 4bb3c2bc-fb41-42d9-812e-19e2ddf04896 2021-07-26T06:15:14.153 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:15:14.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDyUv5gSojCCRAAVN8cJQlPsQWuF5ucA4kaLQ== 2021-07-26T06:15:14.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDyUv5gSojCCRAAVN8cJQlPsQWuF5ucA4kaLQ=="}' 2021-07-26T06:15:14.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4bb3c2bc-fb41-42d9-812e-19e2ddf04896 -i td/osd-scrub-dump/0/new.json 2021-07-26T06:15:14.487 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:15:14.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/0/new.json 2021-07-26T06:15:14.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/0 --osd-journal=td/osd-scrub-dump/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDyUv5gSojCCRAAVN8cJQlPsQWuF5ucA4kaLQ== --osd-uuid 4bb3c2bc-fb41-42d9-812e-19e2ddf04896 2021-07-26T06:15:14.524 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:14.525+0000 7f952c1ac0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:14.526 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:14.527+0000 7f952c1ac0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:14.527 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:14.528+0000 7f952c1ac0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:14.528 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:14.528+0000 7f952c1ac0c0 -1 bluestore(td/osd-scrub-dump/0/block) _read_bdev_label failed to open td/osd-scrub-dump/0/block: (2) No such file or directory 2021-07-26T06:15:14.528 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:14.528+0000 7f952c1ac0c0 -1 bluestore(td/osd-scrub-dump/0/block) _read_bdev_label failed to open td/osd-scrub-dump/0/block: (2) No such file or directory 2021-07-26T06:15:14.529 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:14.528+0000 7f952c1ac0c0 -1 bluestore(td/osd-scrub-dump/0/block) _read_bdev_label failed to open td/osd-scrub-dump/0/block: (2) No such file or directory 2021-07-26T06:15:14.581 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:14.582+0000 7f952c1ac0c0 -1 bluestore(td/osd-scrub-dump/0) _read_fsid unparsable uuid 2021-07-26T06:15:20.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/0/keyring 2021-07-26T06:15:20.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:15:20.424 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:15:20.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:15:20.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:15:20.780 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:15:20.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:15:20.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/0 --osd-journal=td/osd-scrub-dump/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:15:20.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:15:20.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:15:20.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:15:20.810 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:20.811+0000 7f5f0ab490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:20.812 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:20.813+0000 7f5f0ab490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:20.812 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:20.814+0000 7f5f0ab490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:21.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:15:21.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:15:21.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:15:21.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:15:21.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:15:21.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:21.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:15:21.090 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:15:21.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:21.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:15:21.335 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:21.337+0000 7f5f0ab490c0 -1 Falling back to public interface 2021-07-26T06:15:21.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:22.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:22.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:22.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:15:22.367 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:15:22.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:22.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:15:22.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:23.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:23.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:23.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:15:23.650 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:15:23.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:23.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:15:23.704 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:23.705+0000 7f5f0ab490c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:15:23.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:24.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:24.929 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:15:24.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:24.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:15:24.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:24.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:15:25.209 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/54360,v1:127.0.0.1:6803/54360] [v2:127.0.0.1:6804/54360,v1:127.0.0.1:6805/54360] exists,up 4bb3c2bc-fb41-42d9-812e-19e2ddf04896 2021-07-26T06:15:25.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:15:25.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:15:25.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:15:25.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:25.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: run_osd td/osd-scrub-dump 1 2021-07-26T06:15:25.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2021-07-26T06:15:25.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:15:25.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:15:25.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:15:25.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/1 2021-07-26T06:15:25.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 ' 2021-07-26T06:15:25.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:15:25.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:15:25.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:15:25.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/1' 2021-07-26T06:15:25.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/1/journal' 2021-07-26T06:15:25.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:15:25.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:15:25.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:15:25.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2021-07-26T06:15:25.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:15:25.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:15:25.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:15:25.220 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:15:25.220 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:25.220 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:15:25.220 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:15:25.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:15:25.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:15:25.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:15:25.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:15:25.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2021-07-26T06:15:25.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2021-07-26T06:15:25.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:15:25.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:15:25.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:15:25.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:15:25.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:15:25.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/1 2021-07-26T06:15:25.223 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:15:25.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b264c774-d4b2-4843-88af-f6fda9e45141 2021-07-26T06:15:25.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b264c774-d4b2-4843-88af-f6fda9e45141' 2021-07-26T06:15:25.224 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 b264c774-d4b2-4843-88af-f6fda9e45141 2021-07-26T06:15:25.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:15:25.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD9Uv5ghBvwDRAAqvrkImm1TKtK/hvkxmPsmg== 2021-07-26T06:15:25.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD9Uv5ghBvwDRAAqvrkImm1TKtK/hvkxmPsmg=="}' 2021-07-26T06:15:25.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b264c774-d4b2-4843-88af-f6fda9e45141 -i td/osd-scrub-dump/1/new.json 2021-07-26T06:15:25.615 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:15:25.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/1/new.json 2021-07-26T06:15:25.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/1 --osd-journal=td/osd-scrub-dump/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD9Uv5ghBvwDRAAqvrkImm1TKtK/hvkxmPsmg== --osd-uuid b264c774-d4b2-4843-88af-f6fda9e45141 2021-07-26T06:15:25.659 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:25.660+0000 7f12c0fd80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:25.660 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:25.662+0000 7f12c0fd80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:25.661 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:25.662+0000 7f12c0fd80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:25.662 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:25.663+0000 7f12c0fd80c0 -1 bluestore(td/osd-scrub-dump/1/block) _read_bdev_label failed to open td/osd-scrub-dump/1/block: (2) No such file or directory 2021-07-26T06:15:25.662 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:25.663+0000 7f12c0fd80c0 -1 bluestore(td/osd-scrub-dump/1/block) _read_bdev_label failed to open td/osd-scrub-dump/1/block: (2) No such file or directory 2021-07-26T06:15:25.662 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:25.663+0000 7f12c0fd80c0 -1 bluestore(td/osd-scrub-dump/1/block) _read_bdev_label failed to open td/osd-scrub-dump/1/block: (2) No such file or directory 2021-07-26T06:15:25.718 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:25.719+0000 7f12c0fd80c0 -1 bluestore(td/osd-scrub-dump/1) _read_fsid unparsable uuid 2021-07-26T06:15:31.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/1/keyring 2021-07-26T06:15:31.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:15:31.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:15:31.490 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:15:31.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:15:31.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:15:31.839 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:15:31.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:15:31.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/1 --osd-journal=td/osd-scrub-dump/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:15:31.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:15:31.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:15:31.868 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:31.868+0000 7ff4ebd9c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:31.869 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:31.870+0000 7ff4ebd9c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:31.870 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:31.871+0000 7ff4ebd9c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:32.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:15:32.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:15:32.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:15:32.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:15:32.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:15:32.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:32.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:15:32.157 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:15:32.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:32.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:15:32.388 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:32.388+0000 7ff4ebd9c0c0 -1 Falling back to public interface 2021-07-26T06:15:32.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:33.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:33.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:33.484 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:15:33.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:15:33.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:33.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:15:33.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:34.724 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:34.724+0000 7ff4ebd9c0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:15:34.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:34.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:34.769 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:15:34.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:15:34.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:34.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:15:35.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:36.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:36.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:36.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:15:36.055 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:15:36.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:36.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:15:36.335 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/55759,v1:127.0.0.1:6811/55759] [v2:127.0.0.1:6812/55759,v1:127.0.0.1:6813/55759] exists,up b264c774-d4b2-4843-88af-f6fda9e45141 2021-07-26T06:15:36.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:15:36.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:15:36.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:15:36.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:36.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: run_osd td/osd-scrub-dump 2 2021-07-26T06:15:36.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2021-07-26T06:15:36.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:15:36.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:15:36.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:15:36.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/2 2021-07-26T06:15:36.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 ' 2021-07-26T06:15:36.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:15:36.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:15:36.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:15:36.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/2' 2021-07-26T06:15:36.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/2/journal' 2021-07-26T06:15:36.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:15:36.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:15:36.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:15:36.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2021-07-26T06:15:36.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:15:36.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:15:36.347 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:15:36.347 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:15:36.347 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:36.347 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:15:36.348 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:15:36.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:15:36.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:15:36.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:15:36.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:15:36.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2021-07-26T06:15:36.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2021-07-26T06:15:36.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:15:36.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:15:36.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:15:36.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:15:36.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:15:36.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/2 2021-07-26T06:15:36.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:15:36.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ab881f98-eb25-411a-b9da-ba6b8294d1ce 2021-07-26T06:15:36.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ab881f98-eb25-411a-b9da-ba6b8294d1ce' 2021-07-26T06:15:36.351 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 ab881f98-eb25-411a-b9da-ba6b8294d1ce 2021-07-26T06:15:36.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:15:36.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAIU/5gu2yLFRAAiZadbOTcAHt4UzLGL6ElrA== 2021-07-26T06:15:36.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAIU/5gu2yLFRAAiZadbOTcAHt4UzLGL6ElrA=="}' 2021-07-26T06:15:36.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ab881f98-eb25-411a-b9da-ba6b8294d1ce -i td/osd-scrub-dump/2/new.json 2021-07-26T06:15:36.718 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:15:36.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/2/new.json 2021-07-26T06:15:36.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/2 --osd-journal=td/osd-scrub-dump/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAIU/5gu2yLFRAAiZadbOTcAHt4UzLGL6ElrA== --osd-uuid ab881f98-eb25-411a-b9da-ba6b8294d1ce 2021-07-26T06:15:36.761 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:36.761+0000 7f6ae790c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:36.764 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:36.765+0000 7f6ae790c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:36.765 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:36.765+0000 7f6ae790c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:36.766 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:36.766+0000 7f6ae790c0c0 -1 bluestore(td/osd-scrub-dump/2/block) _read_bdev_label failed to open td/osd-scrub-dump/2/block: (2) No such file or directory 2021-07-26T06:15:36.766 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:36.766+0000 7f6ae790c0c0 -1 bluestore(td/osd-scrub-dump/2/block) _read_bdev_label failed to open td/osd-scrub-dump/2/block: (2) No such file or directory 2021-07-26T06:15:36.766 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:36.766+0000 7f6ae790c0c0 -1 bluestore(td/osd-scrub-dump/2/block) _read_bdev_label failed to open td/osd-scrub-dump/2/block: (2) No such file or directory 2021-07-26T06:15:36.825 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:36.826+0000 7f6ae790c0c0 -1 bluestore(td/osd-scrub-dump/2) _read_fsid unparsable uuid 2021-07-26T06:15:42.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/2/keyring 2021-07-26T06:15:42.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:15:42.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:15:42.848 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:15:42.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:15:43.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:15:43.196 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:15:43.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/2 --osd-journal=td/osd-scrub-dump/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:15:43.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:15:43.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:15:43.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:15:43.226 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:43.226+0000 7fa7024270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:43.227 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:43.227+0000 7fa7024270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:43.228 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:43.228+0000 7fa7024270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:43.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:15:43.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:15:43.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:15:43.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:15:43.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:15:43.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:43.493 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:15:43.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:15:43.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:43.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:15:43.763 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:43.762+0000 7fa7024270c0 -1 Falling back to public interface 2021-07-26T06:15:43.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:44.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:44.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:44.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:15:44.766 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:15:44.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:44.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:15:45.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:46.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:46.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:46.052 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:15:46.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:15:46.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:46.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:15:46.159 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:46.159+0000 7fa7024270c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:15:46.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:47.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:47.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:47.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:15:47.368 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:15:47.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:47.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:15:47.694 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/57158,v1:127.0.0.1:6819/57158] [v2:127.0.0.1:6820/57158,v1:127.0.0.1:6821/57158] exists,up ab881f98-eb25-411a-b9da-ba6b8294d1ce 2021-07-26T06:15:47.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:15:47.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:15:47.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:15:47.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:47.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: run_osd td/osd-scrub-dump 3 2021-07-26T06:15:47.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2021-07-26T06:15:47.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:15:47.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T06:15:47.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:15:47.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/3 2021-07-26T06:15:47.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 ' 2021-07-26T06:15:47.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:15:47.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:15:47.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:15:47.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/3' 2021-07-26T06:15:47.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/3/journal' 2021-07-26T06:15:47.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:15:47.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:15:47.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:15:47.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2021-07-26T06:15:47.704 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:15:47.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:15:47.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:15:47.705 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:15:47.705 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:47.705 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:15:47.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:15:47.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:15:47.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:15:47.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:15:47.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:15:47.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2021-07-26T06:15:47.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2021-07-26T06:15:47.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:15:47.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:15:47.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:15:47.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:15:47.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:15:47.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/3 2021-07-26T06:15:47.712 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:15:47.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=047d9d58-6648-40ed-9b07-4c6df838e168 2021-07-26T06:15:47.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 047d9d58-6648-40ed-9b07-4c6df838e168' 2021-07-26T06:15:47.713 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 047d9d58-6648-40ed-9b07-4c6df838e168 2021-07-26T06:15:47.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:15:47.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQATU/5gLZnwKhAA/WHaUjQpwGWIDjuHi3LPiA== 2021-07-26T06:15:47.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQATU/5gLZnwKhAA/WHaUjQpwGWIDjuHi3LPiA=="}' 2021-07-26T06:15:47.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 047d9d58-6648-40ed-9b07-4c6df838e168 -i td/osd-scrub-dump/3/new.json 2021-07-26T06:15:48.094 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:15:48.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/3/new.json 2021-07-26T06:15:48.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/3 --osd-journal=td/osd-scrub-dump/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQATU/5gLZnwKhAA/WHaUjQpwGWIDjuHi3LPiA== --osd-uuid 047d9d58-6648-40ed-9b07-4c6df838e168 2021-07-26T06:15:48.138 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:48.138+0000 7f1e75c9d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:48.152 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:48.152+0000 7f1e75c9d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:48.153 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:48.152+0000 7f1e75c9d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:48.153 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:48.153+0000 7f1e75c9d0c0 -1 bluestore(td/osd-scrub-dump/3/block) _read_bdev_label failed to open td/osd-scrub-dump/3/block: (2) No such file or directory 2021-07-26T06:15:48.153 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:48.153+0000 7f1e75c9d0c0 -1 bluestore(td/osd-scrub-dump/3/block) _read_bdev_label failed to open td/osd-scrub-dump/3/block: (2) No such file or directory 2021-07-26T06:15:48.154 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:48.153+0000 7f1e75c9d0c0 -1 bluestore(td/osd-scrub-dump/3/block) _read_bdev_label failed to open td/osd-scrub-dump/3/block: (2) No such file or directory 2021-07-26T06:15:48.225 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:48.225+0000 7f1e75c9d0c0 -1 bluestore(td/osd-scrub-dump/3) _read_fsid unparsable uuid 2021-07-26T06:15:54.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/3/keyring 2021-07-26T06:15:54.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:15:54.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T06:15:54.279 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd3 key to auth repository 2021-07-26T06:15:54.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:15:54.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T06:15:54.622 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.3 2021-07-26T06:15:54.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:15:54.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/3 --osd-journal=td/osd-scrub-dump/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:15:54.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:15:54.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:15:54.651 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:54.652+0000 7fe241af00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:54.653 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:54.654+0000 7fe241af00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:54.654 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:54.655+0000 7fe241af00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:54.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T06:15:54.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:15:54.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:15:54.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:15:54.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:15:54.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:54.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:15:54.930 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:15:54.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:54.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:15:55.172 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:55.174+0000 7fe241af00c0 -1 Falling back to public interface 2021-07-26T06:15:55.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:56.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:56.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:56.292 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:15:56.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:15:56.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:56.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:15:56.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:57.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:57.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:57.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:15:57.578 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:15:57.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:57.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:15:57.632 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:57.633+0000 7fe241af00c0 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T06:15:57.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:58.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:58.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:58.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:15:58.889 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:15:58.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:58.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:15:59.185 INFO:tasks.workunit.client.0.smithi071.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/58557,v1:127.0.0.1:6827/58557] [v2:127.0.0.1:6828/58557,v1:127.0.0.1:6829/58557] exists,up 047d9d58-6648-40ed-9b07-4c6df838e168 2021-07-26T06:15:59.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:15:59.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:15:59.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:15:59.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:15:59.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: run_osd td/osd-scrub-dump 4 2021-07-26T06:15:59.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2021-07-26T06:15:59.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:15:59.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-07-26T06:15:59.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:15:59.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/4 2021-07-26T06:15:59.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 ' 2021-07-26T06:15:59.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:15:59.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:15:59.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:15:59.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/4' 2021-07-26T06:15:59.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/4/journal' 2021-07-26T06:15:59.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:15:59.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:15:59.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:15:59.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2021-07-26T06:15:59.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:15:59.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:15:59.197 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:15:59.197 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:15:59.198 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:59.198 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:15:59.199 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:15:59.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:15:59.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:15:59.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:15:59.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:15:59.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2021-07-26T06:15:59.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2021-07-26T06:15:59.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:15:59.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:15:59.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:15:59.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:15:59.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:15:59.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/4 2021-07-26T06:15:59.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:15:59.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3a07c832-9ef2-4abe-b368-ef2421a8d3fd 2021-07-26T06:15:59.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 3a07c832-9ef2-4abe-b368-ef2421a8d3fd' 2021-07-26T06:15:59.206 INFO:tasks.workunit.client.0.smithi071.stdout:add osd4 3a07c832-9ef2-4abe-b368-ef2421a8d3fd 2021-07-26T06:15:59.207 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:15:59.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAfU/5gm+WTDBAARg787JqiNhcj3WmMTgPvrA== 2021-07-26T06:15:59.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAfU/5gm+WTDBAARg787JqiNhcj3WmMTgPvrA=="}' 2021-07-26T06:15:59.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3a07c832-9ef2-4abe-b368-ef2421a8d3fd -i td/osd-scrub-dump/4/new.json 2021-07-26T06:15:59.568 INFO:tasks.workunit.client.0.smithi071.stdout:4 2021-07-26T06:15:59.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/4/new.json 2021-07-26T06:15:59.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/4 --osd-journal=td/osd-scrub-dump/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAfU/5gm+WTDBAARg787JqiNhcj3WmMTgPvrA== --osd-uuid 3a07c832-9ef2-4abe-b368-ef2421a8d3fd 2021-07-26T06:15:59.611 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:59.612+0000 7fe3e389a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:59.622 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:59.623+0000 7fe3e389a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:59.623 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:59.624+0000 7fe3e389a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:59.624 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:59.625+0000 7fe3e389a0c0 -1 bluestore(td/osd-scrub-dump/4/block) _read_bdev_label failed to open td/osd-scrub-dump/4/block: (2) No such file or directory 2021-07-26T06:15:59.624 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:59.625+0000 7fe3e389a0c0 -1 bluestore(td/osd-scrub-dump/4/block) _read_bdev_label failed to open td/osd-scrub-dump/4/block: (2) No such file or directory 2021-07-26T06:15:59.624 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:59.625+0000 7fe3e389a0c0 -1 bluestore(td/osd-scrub-dump/4/block) _read_bdev_label failed to open td/osd-scrub-dump/4/block: (2) No such file or directory 2021-07-26T06:15:59.703 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:15:59.704+0000 7fe3e389a0c0 -1 bluestore(td/osd-scrub-dump/4) _read_fsid unparsable uuid 2021-07-26T06:16:05.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/4/keyring 2021-07-26T06:16:05.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:16:05.716 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd4 key to auth repository 2021-07-26T06:16:05.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2021-07-26T06:16:05.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:16:06.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2021-07-26T06:16:06.062 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.4 2021-07-26T06:16:06.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/4 --osd-journal=td/osd-scrub-dump/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:16:06.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:16:06.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:16:06.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:16:06.091 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:06.092+0000 7f7c7d31f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:06.093 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:06.094+0000 7f7c7d31f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:06.093 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:06.095+0000 7f7c7d31f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:06.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2021-07-26T06:16:06.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:16:06.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=4 2021-07-26T06:16:06.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:16:06.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:16:06.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:06.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:16:06.385 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:16:06.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:06.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:16:06.607 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:06.609+0000 7f7c7d31f0c0 -1 Falling back to public interface 2021-07-26T06:16:06.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:16:07.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:16:07.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:07.725 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:16:07.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:16:07.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:07.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:16:08.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:16:09.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:16:09.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:09.010 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:16:09.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:16:09.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:09.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:16:09.098 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:09.098+0000 7f7c7d31f0c0 -1 osd.4 0 log_to_monitors {default=true} 2021-07-26T06:16:09.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:16:10.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:16:10.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:10.331 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:16:10.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:16:10.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:10.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:16:10.659 INFO:tasks.workunit.client.0.smithi071.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/59956,v1:127.0.0.1:6835/59956] [v2:127.0.0.1:6836/59956,v1:127.0.0.1:6837/59956] exists,up 3a07c832-9ef2-4abe-b368-ef2421a8d3fd 2021-07-26T06:16:10.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:16:10.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:16:10.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:16:10.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:16:10.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: run_osd td/osd-scrub-dump 5 2021-07-26T06:16:10.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2021-07-26T06:16:10.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:16:10.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2021-07-26T06:16:10.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:16:10.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/5 2021-07-26T06:16:10.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 ' 2021-07-26T06:16:10.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:16:10.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:16:10.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:16:10.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/5' 2021-07-26T06:16:10.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/5/journal' 2021-07-26T06:16:10.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:16:10.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:16:10.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:16:10.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2021-07-26T06:16:10.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:16:10.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:16:10.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:16:10.669 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:16:10.670 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:16:10.670 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:16:10.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:16:10.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' 2021-07-26T06:16:10.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:16:10.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:16:10.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:16:10.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2021-07-26T06:16:10.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2021-07-26T06:16:10.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:16:10.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:16:10.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:16:10.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:16:10.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:16:10.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/5 2021-07-26T06:16:10.673 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:16:10.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0449579d-0e49-4ab8-b01c-17d90cfb0b30 2021-07-26T06:16:10.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 0449579d-0e49-4ab8-b01c-17d90cfb0b30' 2021-07-26T06:16:10.675 INFO:tasks.workunit.client.0.smithi071.stdout:add osd5 0449579d-0e49-4ab8-b01c-17d90cfb0b30 2021-07-26T06:16:10.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:16:10.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAqU/5gJj7QKBAA6t0xADGd8z333DHcEAg4lA== 2021-07-26T06:16:10.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAqU/5gJj7QKBAA6t0xADGd8z333DHcEAg4lA=="}' 2021-07-26T06:16:10.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0449579d-0e49-4ab8-b01c-17d90cfb0b30 -i td/osd-scrub-dump/5/new.json 2021-07-26T06:16:11.085 INFO:tasks.workunit.client.0.smithi071.stdout:5 2021-07-26T06:16:11.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/5/new.json 2021-07-26T06:16:11.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/5 --osd-journal=td/osd-scrub-dump/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAqU/5gJj7QKBAA6t0xADGd8z333DHcEAg4lA== --osd-uuid 0449579d-0e49-4ab8-b01c-17d90cfb0b30 2021-07-26T06:16:11.129 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:11.130+0000 7fc3e31910c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:11.130 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:11.132+0000 7fc3e31910c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:11.131 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:11.132+0000 7fc3e31910c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:11.132 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:11.133+0000 7fc3e31910c0 -1 bluestore(td/osd-scrub-dump/5/block) _read_bdev_label failed to open td/osd-scrub-dump/5/block: (2) No such file or directory 2021-07-26T06:16:11.132 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:11.133+0000 7fc3e31910c0 -1 bluestore(td/osd-scrub-dump/5/block) _read_bdev_label failed to open td/osd-scrub-dump/5/block: (2) No such file or directory 2021-07-26T06:16:11.132 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:11.133+0000 7fc3e31910c0 -1 bluestore(td/osd-scrub-dump/5/block) _read_bdev_label failed to open td/osd-scrub-dump/5/block: (2) No such file or directory 2021-07-26T06:16:11.223 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:11.224+0000 7fc3e31910c0 -1 bluestore(td/osd-scrub-dump/5) _read_fsid unparsable uuid 2021-07-26T06:16:17.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/5/keyring 2021-07-26T06:16:17.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:16:17.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2021-07-26T06:16:17.306 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd5 key to auth repository 2021-07-26T06:16:17.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:16:17.655 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.5 2021-07-26T06:16:17.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2021-07-26T06:16:17.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=65f0d09a-f9e5-4a74-803e-9a146e81426a --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/5 --osd-journal=td/osd-scrub-dump/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.53231/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:16:17.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:16:17.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:16:17.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:16:17.685 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:17.686+0000 7fd297c1a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:17.687 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:17.688+0000 7fd297c1a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:17.687 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:17.689+0000 7fd297c1a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:17.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2021-07-26T06:16:17.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:16:17.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:16:17.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:16:17.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:16:17.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:17.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:16:17.977 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:16:17.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:17.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:16:18.207 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:18.207+0000 7fd297c1a0c0 -1 Falling back to public interface 2021-07-26T06:16:18.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:16:19.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:16:19.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:19.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:16:19.290 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:16:19.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:19.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:16:19.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:16:20.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:16:20.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:20.573 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:16:20.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:16:20.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:20.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:16:20.577 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:16:20.578+0000 7fd297c1a0c0 -1 osd.5 0 log_to_monitors {default=true} 2021-07-26T06:16:20.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:16:21.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:16:21.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:21.868 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:16:21.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:16:21.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:21.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:16:22.145 INFO:tasks.workunit.client.0.smithi071.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/61355,v1:127.0.0.1:6843/61355] [v2:127.0.0.1:6844/61355,v1:127.0.0.1:6845/61355] exists,up 0449579d-0e49-4ab8-b01c-17d90cfb0b30 2021-07-26T06:16:22.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:16:22.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:16:22.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:16:22.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:64: TEST_recover_unexpected: seq 1 3 2021-07-26T06:16:22.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:64: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2021-07-26T06:16:22.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: create_pool test1 16 16 2021-07-26T06:16:22.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 16 16 2021-07-26T06:16:22.541 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'test1' created 2021-07-26T06:16:22.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:16:23.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:64: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2021-07-26T06:16:23.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: create_pool test2 16 16 2021-07-26T06:16:23.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 16 16 2021-07-26T06:16:24.835 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'test2' created 2021-07-26T06:16:24.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:16:25.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:64: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2021-07-26T06:16:25.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: create_pool test3 16 16 2021-07-26T06:16:25.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 16 16 2021-07-26T06:16:27.065 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'test3' created 2021-07-26T06:16:27.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:16:28.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:69: TEST_recover_unexpected: wait_for_clean 2021-07-26T06:16:28.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:16:28.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:16:28.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:16:28.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:16:28.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:16:28.081 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:16:28.081 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:16:28.081 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:16:28.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:16:28.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:16:28.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:16:28.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:16:28.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:16:28.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:16:28.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:16:28.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:16:28.556 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:16:28.556 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:16:28.557 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:16:28.557 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:16:28.557 INFO:tasks.workunit.client.0.smithi071.stderr:5' 2021-07-26T06:16:28.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:16:28.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:16:28.558 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:16:28.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836494 2021-07-26T06:16:28.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836494 2021-07-26T06:16:28.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494' 2021-07-26T06:16:28.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:16:28.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:16:29.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672972 2021-07-26T06:16:29.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672972 2021-07-26T06:16:29.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672972' 2021-07-26T06:16:29.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:16:29.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:16:29.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509451 2021-07-26T06:16:29.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509451 2021-07-26T06:16:29.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509451' 2021-07-26T06:16:29.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:16:29.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:16:29.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345928 2021-07-26T06:16:29.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345928 2021-07-26T06:16:29.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509451 3-85899345928' 2021-07-26T06:16:29.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:16:29.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:16:29.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182405 2021-07-26T06:16:29.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182405 2021-07-26T06:16:29.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509451 3-85899345928 4-107374182405' 2021-07-26T06:16:29.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:16:29.450 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:16:29.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018883 2021-07-26T06:16:29.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018883 2021-07-26T06:16:29.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509451 3-85899345928 4-107374182405 5-128849018883' 2021-07-26T06:16:29.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:16:29.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836494 2021-07-26T06:16:29.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:16:29.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:16:29.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836494 2021-07-26T06:16:29.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:16:29.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836494 2021-07-26T06:16:29.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2021-07-26T06:16:29.582 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836494 2021-07-26T06:16:29.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:16:29.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836494 -lt 21474836494 2021-07-26T06:16:29.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:16:29.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672972 2021-07-26T06:16:29.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:16:29.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:16:29.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672972 2021-07-26T06:16:29.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:16:29.859 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672972 2021-07-26T06:16:29.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672972 2021-07-26T06:16:29.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2021-07-26T06:16:29.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:16:30.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672971 -lt 42949672972 2021-07-26T06:16:30.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:16:31.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:16:31.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:16:31.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672972 -lt 42949672972 2021-07-26T06:16:31.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:16:31.431 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509451 2021-07-26T06:16:31.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:16:31.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:16:31.433 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509451 2021-07-26T06:16:31.433 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:16:31.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509451 2021-07-26T06:16:31.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2021-07-26T06:16:31.435 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509451 2021-07-26T06:16:31.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:16:31.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509451 -lt 64424509451 2021-07-26T06:16:31.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:16:31.733 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345928 2021-07-26T06:16:31.734 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:16:31.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:16:31.734 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345928 2021-07-26T06:16:31.734 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:16:31.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345928 2021-07-26T06:16:31.735 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.3 seq 85899345928 2021-07-26T06:16:31.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2021-07-26T06:16:31.737 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:16:32.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345928 -lt 85899345928 2021-07-26T06:16:32.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:16:32.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182405 2021-07-26T06:16:32.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:16:32.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:16:32.042 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182405 2021-07-26T06:16:32.043 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:16:32.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182405 2021-07-26T06:16:32.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2021-07-26T06:16:32.044 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.4 seq 107374182405 2021-07-26T06:16:32.045 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:16:32.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182406 -lt 107374182405 2021-07-26T06:16:32.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:16:32.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018883 2021-07-26T06:16:32.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:16:32.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:16:32.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018883 2021-07-26T06:16:32.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:16:32.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018883 2021-07-26T06:16:32.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018883' 2021-07-26T06:16:32.347 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.5 seq 128849018883 2021-07-26T06:16:32.347 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:16:32.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018883 -lt 128849018883 2021-07-26T06:16:32.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:16:32.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:16:32.652 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:16:33.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 48 == 0 2021-07-26T06:16:33.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:16:33.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:16:33.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:16:33.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:16:33.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:16:33.042 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:16:33.042 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:16:33.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=48 2021-07-26T06:16:33.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:16:33.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:16:33.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:16:33.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 48 = 48 2021-07-26T06:16:33.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:16:33.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:16:33.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:71: TEST_recover_unexpected: dd if=/dev/urandom of=datafile bs=4k count=2 2021-07-26T06:16:33.707 INFO:tasks.workunit.client.0.smithi071.stderr:2+0 records in 2021-07-26T06:16:33.707 INFO:tasks.workunit.client.0.smithi071.stderr:2+0 records out 2021-07-26T06:16:33.708 INFO:tasks.workunit.client.0.smithi071.stderr:8192 bytes (8.2 kB, 8.0 KiB) copied, 7.8979e-05 s, 104 MB/s 2021-07-26T06:16:33.708 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:72: TEST_recover_unexpected: seq 1 3 2021-07-26T06:16:33.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:72: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2021-07-26T06:16:33.709 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: seq 1 1000 2021-07-26T06:16:33.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:33.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj1 datafile 2021-07-26T06:16:33.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:33.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj2 datafile 2021-07-26T06:16:33.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:33.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj3 datafile 2021-07-26T06:16:34.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:34.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj4 datafile 2021-07-26T06:16:34.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:34.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj5 datafile 2021-07-26T06:16:34.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:34.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj6 datafile 2021-07-26T06:16:34.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:34.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj7 datafile 2021-07-26T06:16:34.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:34.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj8 datafile 2021-07-26T06:16:34.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:34.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj9 datafile 2021-07-26T06:16:35.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:35.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj10 datafile 2021-07-26T06:16:35.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:35.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj11 datafile 2021-07-26T06:16:35.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:35.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj12 datafile 2021-07-26T06:16:35.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:35.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj13 datafile 2021-07-26T06:16:35.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:35.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj14 datafile 2021-07-26T06:16:35.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:35.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj15 datafile 2021-07-26T06:16:35.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:35.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj16 datafile 2021-07-26T06:16:36.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:36.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj17 datafile 2021-07-26T06:16:36.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:36.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj18 datafile 2021-07-26T06:16:36.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:36.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj19 datafile 2021-07-26T06:16:36.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:36.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj20 datafile 2021-07-26T06:16:36.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:36.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj21 datafile 2021-07-26T06:16:36.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:36.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj22 datafile 2021-07-26T06:16:36.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:36.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj23 datafile 2021-07-26T06:16:36.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:36.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj24 datafile 2021-07-26T06:16:37.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:37.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj25 datafile 2021-07-26T06:16:37.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:37.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj26 datafile 2021-07-26T06:16:37.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:37.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj27 datafile 2021-07-26T06:16:37.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:37.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj28 datafile 2021-07-26T06:16:37.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:37.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj29 datafile 2021-07-26T06:16:37.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:37.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj30 datafile 2021-07-26T06:16:37.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:37.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj31 datafile 2021-07-26T06:16:38.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:38.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj32 datafile 2021-07-26T06:16:38.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:38.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj33 datafile 2021-07-26T06:16:38.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:38.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj34 datafile 2021-07-26T06:16:38.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:38.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj35 datafile 2021-07-26T06:16:38.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:38.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj36 datafile 2021-07-26T06:16:38.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:38.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj37 datafile 2021-07-26T06:16:38.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:38.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj38 datafile 2021-07-26T06:16:39.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:39.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj39 datafile 2021-07-26T06:16:39.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:39.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj40 datafile 2021-07-26T06:16:39.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:39.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj41 datafile 2021-07-26T06:16:39.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:39.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj42 datafile 2021-07-26T06:16:39.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:39.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj43 datafile 2021-07-26T06:16:39.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:39.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj44 datafile 2021-07-26T06:16:39.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:39.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj45 datafile 2021-07-26T06:16:40.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:40.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj46 datafile 2021-07-26T06:16:40.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:40.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj47 datafile 2021-07-26T06:16:40.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:40.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj48 datafile 2021-07-26T06:16:40.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:40.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj49 datafile 2021-07-26T06:16:40.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:40.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj50 datafile 2021-07-26T06:16:40.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:40.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj51 datafile 2021-07-26T06:16:40.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:40.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj52 datafile 2021-07-26T06:16:40.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:40.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj53 datafile 2021-07-26T06:16:41.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:41.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj54 datafile 2021-07-26T06:16:41.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:41.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj55 datafile 2021-07-26T06:16:41.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:41.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj56 datafile 2021-07-26T06:16:41.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:41.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj57 datafile 2021-07-26T06:16:41.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:41.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj58 datafile 2021-07-26T06:16:41.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:41.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj59 datafile 2021-07-26T06:16:41.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:41.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj60 datafile 2021-07-26T06:16:42.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:42.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj61 datafile 2021-07-26T06:16:42.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:42.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj62 datafile 2021-07-26T06:16:42.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:42.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj63 datafile 2021-07-26T06:16:42.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:42.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj64 datafile 2021-07-26T06:16:42.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:42.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj65 datafile 2021-07-26T06:16:42.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:42.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj66 datafile 2021-07-26T06:16:42.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:42.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj67 datafile 2021-07-26T06:16:43.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:43.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj68 datafile 2021-07-26T06:16:43.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:43.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj69 datafile 2021-07-26T06:16:43.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:43.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj70 datafile 2021-07-26T06:16:43.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:43.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj71 datafile 2021-07-26T06:16:43.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:43.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj72 datafile 2021-07-26T06:16:43.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:43.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj73 datafile 2021-07-26T06:16:43.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:43.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj74 datafile 2021-07-26T06:16:44.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:44.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj75 datafile 2021-07-26T06:16:44.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:44.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj76 datafile 2021-07-26T06:16:44.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:44.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj77 datafile 2021-07-26T06:16:44.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:44.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj78 datafile 2021-07-26T06:16:44.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:44.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj79 datafile 2021-07-26T06:16:44.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:44.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj80 datafile 2021-07-26T06:16:44.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:44.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj81 datafile 2021-07-26T06:16:45.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:45.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj82 datafile 2021-07-26T06:16:45.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:45.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj83 datafile 2021-07-26T06:16:45.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:45.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj84 datafile 2021-07-26T06:16:45.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:45.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj85 datafile 2021-07-26T06:16:45.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:45.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj86 datafile 2021-07-26T06:16:45.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:45.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj87 datafile 2021-07-26T06:16:45.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:45.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj88 datafile 2021-07-26T06:16:46.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:46.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj89 datafile 2021-07-26T06:16:46.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:46.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj90 datafile 2021-07-26T06:16:46.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:46.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj91 datafile 2021-07-26T06:16:46.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:46.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj92 datafile 2021-07-26T06:16:46.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:46.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj93 datafile 2021-07-26T06:16:46.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:46.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj94 datafile 2021-07-26T06:16:46.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:46.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj95 datafile 2021-07-26T06:16:47.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:47.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj96 datafile 2021-07-26T06:16:47.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:47.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj97 datafile 2021-07-26T06:16:47.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:47.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj98 datafile 2021-07-26T06:16:47.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:47.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj99 datafile 2021-07-26T06:16:47.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:47.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj100 datafile 2021-07-26T06:16:47.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:47.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj101 datafile 2021-07-26T06:16:47.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:47.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj102 datafile 2021-07-26T06:16:48.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:48.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj103 datafile 2021-07-26T06:16:48.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:48.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj104 datafile 2021-07-26T06:16:48.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:48.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj105 datafile 2021-07-26T06:16:48.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:48.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj106 datafile 2021-07-26T06:16:48.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:48.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj107 datafile 2021-07-26T06:16:48.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:48.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj108 datafile 2021-07-26T06:16:49.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:49.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj109 datafile 2021-07-26T06:16:49.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:49.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj110 datafile 2021-07-26T06:16:49.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:49.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj111 datafile 2021-07-26T06:16:49.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:49.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj112 datafile 2021-07-26T06:16:49.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:49.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj113 datafile 2021-07-26T06:16:49.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:49.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj114 datafile 2021-07-26T06:16:49.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:49.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj115 datafile 2021-07-26T06:16:50.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:50.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj116 datafile 2021-07-26T06:16:50.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:50.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj117 datafile 2021-07-26T06:16:50.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:50.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj118 datafile 2021-07-26T06:16:50.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:50.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj119 datafile 2021-07-26T06:16:50.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:50.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj120 datafile 2021-07-26T06:16:50.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:50.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj121 datafile 2021-07-26T06:16:50.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:50.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj122 datafile 2021-07-26T06:16:51.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:51.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj123 datafile 2021-07-26T06:16:51.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:51.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj124 datafile 2021-07-26T06:16:51.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:51.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj125 datafile 2021-07-26T06:16:51.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:51.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj126 datafile 2021-07-26T06:16:51.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:51.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj127 datafile 2021-07-26T06:16:51.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:51.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj128 datafile 2021-07-26T06:16:51.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:51.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj129 datafile 2021-07-26T06:16:52.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:52.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj130 datafile 2021-07-26T06:16:52.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:52.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj131 datafile 2021-07-26T06:16:52.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:52.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj132 datafile 2021-07-26T06:16:52.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:52.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj133 datafile 2021-07-26T06:16:52.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:52.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj134 datafile 2021-07-26T06:16:52.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:52.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj135 datafile 2021-07-26T06:16:52.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:52.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj136 datafile 2021-07-26T06:16:53.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:53.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj137 datafile 2021-07-26T06:16:53.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:53.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj138 datafile 2021-07-26T06:16:53.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:53.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj139 datafile 2021-07-26T06:16:53.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:53.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj140 datafile 2021-07-26T06:16:53.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:53.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj141 datafile 2021-07-26T06:16:53.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:53.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj142 datafile 2021-07-26T06:16:53.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:53.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj143 datafile 2021-07-26T06:16:54.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:54.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj144 datafile 2021-07-26T06:16:54.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:54.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj145 datafile 2021-07-26T06:16:54.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:54.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj146 datafile 2021-07-26T06:16:54.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:54.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj147 datafile 2021-07-26T06:16:54.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:54.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj148 datafile 2021-07-26T06:16:54.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:54.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj149 datafile 2021-07-26T06:16:54.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:54.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj150 datafile 2021-07-26T06:16:55.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:55.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj151 datafile 2021-07-26T06:16:55.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:55.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj152 datafile 2021-07-26T06:16:55.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:55.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj153 datafile 2021-07-26T06:16:55.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:55.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj154 datafile 2021-07-26T06:16:55.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:55.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj155 datafile 2021-07-26T06:16:55.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:55.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj156 datafile 2021-07-26T06:16:55.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:55.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj157 datafile 2021-07-26T06:16:55.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:55.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj158 datafile 2021-07-26T06:16:56.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:56.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj159 datafile 2021-07-26T06:16:56.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:56.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj160 datafile 2021-07-26T06:16:56.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:56.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj161 datafile 2021-07-26T06:16:56.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:56.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj162 datafile 2021-07-26T06:16:56.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:56.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj163 datafile 2021-07-26T06:16:56.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:56.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj164 datafile 2021-07-26T06:16:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:56.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj165 datafile 2021-07-26T06:16:57.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:57.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj166 datafile 2021-07-26T06:16:57.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:57.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj167 datafile 2021-07-26T06:16:57.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:57.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj168 datafile 2021-07-26T06:16:57.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:57.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj169 datafile 2021-07-26T06:16:57.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:57.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj170 datafile 2021-07-26T06:16:57.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:57.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj171 datafile 2021-07-26T06:16:57.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:57.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj172 datafile 2021-07-26T06:16:57.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:57.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj173 datafile 2021-07-26T06:16:58.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:58.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj174 datafile 2021-07-26T06:16:58.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:58.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj175 datafile 2021-07-26T06:16:58.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:58.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj176 datafile 2021-07-26T06:16:58.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:58.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj177 datafile 2021-07-26T06:16:58.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:58.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj178 datafile 2021-07-26T06:16:58.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:58.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj179 datafile 2021-07-26T06:16:59.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:59.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj180 datafile 2021-07-26T06:16:59.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:59.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj181 datafile 2021-07-26T06:16:59.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:59.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj182 datafile 2021-07-26T06:16:59.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:59.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj183 datafile 2021-07-26T06:16:59.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:59.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj184 datafile 2021-07-26T06:16:59.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:59.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj185 datafile 2021-07-26T06:16:59.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:16:59.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj186 datafile 2021-07-26T06:17:00.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:00.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj187 datafile 2021-07-26T06:17:00.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:00.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj188 datafile 2021-07-26T06:17:00.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:00.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj189 datafile 2021-07-26T06:17:00.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:00.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj190 datafile 2021-07-26T06:17:00.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:00.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj191 datafile 2021-07-26T06:17:00.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:00.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj192 datafile 2021-07-26T06:17:00.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:00.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj193 datafile 2021-07-26T06:17:01.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:01.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj194 datafile 2021-07-26T06:17:01.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:01.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj195 datafile 2021-07-26T06:17:01.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:01.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj196 datafile 2021-07-26T06:17:01.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:01.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj197 datafile 2021-07-26T06:17:01.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:01.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj198 datafile 2021-07-26T06:17:01.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:01.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj199 datafile 2021-07-26T06:17:01.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:01.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj200 datafile 2021-07-26T06:17:01.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:01.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj201 datafile 2021-07-26T06:17:02.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:02.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj202 datafile 2021-07-26T06:17:02.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:02.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj203 datafile 2021-07-26T06:17:02.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:02.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj204 datafile 2021-07-26T06:17:02.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:02.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj205 datafile 2021-07-26T06:17:02.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:02.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj206 datafile 2021-07-26T06:17:02.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:02.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj207 datafile 2021-07-26T06:17:02.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:02.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj208 datafile 2021-07-26T06:17:03.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:03.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj209 datafile 2021-07-26T06:17:03.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:03.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj210 datafile 2021-07-26T06:17:03.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:03.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj211 datafile 2021-07-26T06:17:03.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:03.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj212 datafile 2021-07-26T06:17:03.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:03.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj213 datafile 2021-07-26T06:17:03.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:03.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj214 datafile 2021-07-26T06:17:03.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:03.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj215 datafile 2021-07-26T06:17:04.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:04.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj216 datafile 2021-07-26T06:17:04.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:04.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj217 datafile 2021-07-26T06:17:04.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:04.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj218 datafile 2021-07-26T06:17:04.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:04.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj219 datafile 2021-07-26T06:17:04.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:04.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj220 datafile 2021-07-26T06:17:04.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:04.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj221 datafile 2021-07-26T06:17:04.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:04.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj222 datafile 2021-07-26T06:17:05.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:05.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj223 datafile 2021-07-26T06:17:05.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:05.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj224 datafile 2021-07-26T06:17:05.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:05.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj225 datafile 2021-07-26T06:17:05.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:05.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj226 datafile 2021-07-26T06:17:05.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:05.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj227 datafile 2021-07-26T06:17:05.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:05.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj228 datafile 2021-07-26T06:17:05.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:05.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj229 datafile 2021-07-26T06:17:06.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:06.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj230 datafile 2021-07-26T06:17:06.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:06.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj231 datafile 2021-07-26T06:17:06.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:06.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj232 datafile 2021-07-26T06:17:06.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:06.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj233 datafile 2021-07-26T06:17:06.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:06.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj234 datafile 2021-07-26T06:17:06.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:06.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj235 datafile 2021-07-26T06:17:06.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:06.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj236 datafile 2021-07-26T06:17:07.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:07.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj237 datafile 2021-07-26T06:17:07.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:07.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj238 datafile 2021-07-26T06:17:07.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:07.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj239 datafile 2021-07-26T06:17:07.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:07.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj240 datafile 2021-07-26T06:17:07.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:07.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj241 datafile 2021-07-26T06:17:07.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:07.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj242 datafile 2021-07-26T06:17:08.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:08.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj243 datafile 2021-07-26T06:17:08.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:08.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj244 datafile 2021-07-26T06:17:08.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:08.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj245 datafile 2021-07-26T06:17:08.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:08.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj246 datafile 2021-07-26T06:17:08.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:08.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj247 datafile 2021-07-26T06:17:08.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:08.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj248 datafile 2021-07-26T06:17:08.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:08.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj249 datafile 2021-07-26T06:17:09.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:09.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj250 datafile 2021-07-26T06:17:09.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:09.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj251 datafile 2021-07-26T06:17:09.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:09.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj252 datafile 2021-07-26T06:17:09.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:09.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj253 datafile 2021-07-26T06:17:09.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:09.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj254 datafile 2021-07-26T06:17:09.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:09.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj255 datafile 2021-07-26T06:17:09.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:09.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj256 datafile 2021-07-26T06:17:10.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:10.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj257 datafile 2021-07-26T06:17:10.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:10.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj258 datafile 2021-07-26T06:17:10.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:10.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj259 datafile 2021-07-26T06:17:10.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:10.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj260 datafile 2021-07-26T06:17:10.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:10.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj261 datafile 2021-07-26T06:17:10.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:10.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj262 datafile 2021-07-26T06:17:10.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:10.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj263 datafile 2021-07-26T06:17:11.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:11.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj264 datafile 2021-07-26T06:17:11.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:11.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj265 datafile 2021-07-26T06:17:11.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:11.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj266 datafile 2021-07-26T06:17:11.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:11.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj267 datafile 2021-07-26T06:17:11.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:11.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj268 datafile 2021-07-26T06:17:11.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:11.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj269 datafile 2021-07-26T06:17:11.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:11.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj270 datafile 2021-07-26T06:17:12.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:12.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj271 datafile 2021-07-26T06:17:12.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:12.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj272 datafile 2021-07-26T06:17:12.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:12.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj273 datafile 2021-07-26T06:17:12.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:12.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj274 datafile 2021-07-26T06:17:12.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:12.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj275 datafile 2021-07-26T06:17:12.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:12.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj276 datafile 2021-07-26T06:17:12.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:12.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj277 datafile 2021-07-26T06:17:13.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:13.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj278 datafile 2021-07-26T06:17:13.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:13.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj279 datafile 2021-07-26T06:17:13.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:13.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj280 datafile 2021-07-26T06:17:13.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:13.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj281 datafile 2021-07-26T06:17:13.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:13.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj282 datafile 2021-07-26T06:17:13.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:13.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj283 datafile 2021-07-26T06:17:13.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:13.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj284 datafile 2021-07-26T06:17:14.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:14.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj285 datafile 2021-07-26T06:17:14.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:14.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj286 datafile 2021-07-26T06:17:14.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:14.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj287 datafile 2021-07-26T06:17:14.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:14.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj288 datafile 2021-07-26T06:17:14.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:14.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj289 datafile 2021-07-26T06:17:14.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:14.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj290 datafile 2021-07-26T06:17:14.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:14.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj291 datafile 2021-07-26T06:17:15.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:15.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj292 datafile 2021-07-26T06:17:15.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:15.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj293 datafile 2021-07-26T06:17:15.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:15.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj294 datafile 2021-07-26T06:17:15.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:15.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj295 datafile 2021-07-26T06:17:15.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:15.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj296 datafile 2021-07-26T06:17:15.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:15.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj297 datafile 2021-07-26T06:17:16.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:16.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj298 datafile 2021-07-26T06:17:16.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:16.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj299 datafile 2021-07-26T06:17:16.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:16.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj300 datafile 2021-07-26T06:17:16.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:16.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj301 datafile 2021-07-26T06:17:16.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:16.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj302 datafile 2021-07-26T06:17:16.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:16.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj303 datafile 2021-07-26T06:17:16.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:16.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj304 datafile 2021-07-26T06:17:16.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:16.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj305 datafile 2021-07-26T06:17:17.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:17.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj306 datafile 2021-07-26T06:17:17.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:17.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj307 datafile 2021-07-26T06:17:17.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:17.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj308 datafile 2021-07-26T06:17:17.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:17.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj309 datafile 2021-07-26T06:17:17.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:17.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj310 datafile 2021-07-26T06:17:17.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:17.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj311 datafile 2021-07-26T06:17:18.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:18.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj312 datafile 2021-07-26T06:17:18.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:18.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj313 datafile 2021-07-26T06:17:18.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:18.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj314 datafile 2021-07-26T06:17:18.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:18.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj315 datafile 2021-07-26T06:17:18.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:18.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj316 datafile 2021-07-26T06:17:18.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:18.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj317 datafile 2021-07-26T06:17:18.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:18.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj318 datafile 2021-07-26T06:17:19.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:19.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj319 datafile 2021-07-26T06:17:19.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:19.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj320 datafile 2021-07-26T06:17:19.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:19.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj321 datafile 2021-07-26T06:17:19.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:19.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj322 datafile 2021-07-26T06:17:19.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:19.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj323 datafile 2021-07-26T06:17:20.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:20.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj324 datafile 2021-07-26T06:17:20.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:20.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj325 datafile 2021-07-26T06:17:20.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:20.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj326 datafile 2021-07-26T06:17:20.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:20.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj327 datafile 2021-07-26T06:17:20.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:20.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj328 datafile 2021-07-26T06:17:20.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:20.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj329 datafile 2021-07-26T06:17:20.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:20.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj330 datafile 2021-07-26T06:17:20.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:20.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj331 datafile 2021-07-26T06:17:21.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:21.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj332 datafile 2021-07-26T06:17:21.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:21.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj333 datafile 2021-07-26T06:17:21.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:21.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj334 datafile 2021-07-26T06:17:21.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:21.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj335 datafile 2021-07-26T06:17:21.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:21.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj336 datafile 2021-07-26T06:17:21.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:21.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj337 datafile 2021-07-26T06:17:22.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:22.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj338 datafile 2021-07-26T06:17:22.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:22.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj339 datafile 2021-07-26T06:17:22.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:22.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj340 datafile 2021-07-26T06:17:22.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:22.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj341 datafile 2021-07-26T06:17:22.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:22.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj342 datafile 2021-07-26T06:17:22.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:22.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj343 datafile 2021-07-26T06:17:22.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:22.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj344 datafile 2021-07-26T06:17:22.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:22.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj345 datafile 2021-07-26T06:17:23.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:23.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj346 datafile 2021-07-26T06:17:23.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:23.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj347 datafile 2021-07-26T06:17:23.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:23.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj348 datafile 2021-07-26T06:17:23.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:23.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj349 datafile 2021-07-26T06:17:23.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:23.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj350 datafile 2021-07-26T06:17:23.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:23.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj351 datafile 2021-07-26T06:17:23.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:23.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj352 datafile 2021-07-26T06:17:24.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:24.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj353 datafile 2021-07-26T06:17:24.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:24.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj354 datafile 2021-07-26T06:17:24.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:24.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj355 datafile 2021-07-26T06:17:24.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:24.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj356 datafile 2021-07-26T06:17:24.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:24.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj357 datafile 2021-07-26T06:17:24.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:24.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj358 datafile 2021-07-26T06:17:24.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:24.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj359 datafile 2021-07-26T06:17:25.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:25.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj360 datafile 2021-07-26T06:17:25.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:25.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj361 datafile 2021-07-26T06:17:25.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:25.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj362 datafile 2021-07-26T06:17:25.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:25.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj363 datafile 2021-07-26T06:17:25.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:25.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj364 datafile 2021-07-26T06:17:25.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:25.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj365 datafile 2021-07-26T06:17:25.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:25.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj366 datafile 2021-07-26T06:17:26.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:26.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj367 datafile 2021-07-26T06:17:26.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:26.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj368 datafile 2021-07-26T06:17:26.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:26.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj369 datafile 2021-07-26T06:17:26.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:26.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj370 datafile 2021-07-26T06:17:26.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:26.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj371 datafile 2021-07-26T06:17:26.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:26.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj372 datafile 2021-07-26T06:17:26.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:26.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj373 datafile 2021-07-26T06:17:27.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:27.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj374 datafile 2021-07-26T06:17:27.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:27.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj375 datafile 2021-07-26T06:17:27.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:27.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj376 datafile 2021-07-26T06:17:27.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:27.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj377 datafile 2021-07-26T06:17:27.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:27.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj378 datafile 2021-07-26T06:17:27.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:27.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj379 datafile 2021-07-26T06:17:27.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:27.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj380 datafile 2021-07-26T06:17:28.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:28.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj381 datafile 2021-07-26T06:17:28.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:28.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj382 datafile 2021-07-26T06:17:28.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:28.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj383 datafile 2021-07-26T06:17:28.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:28.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj384 datafile 2021-07-26T06:17:28.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:28.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj385 datafile 2021-07-26T06:17:28.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:28.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj386 datafile 2021-07-26T06:17:28.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:28.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj387 datafile 2021-07-26T06:17:29.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:29.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj388 datafile 2021-07-26T06:17:29.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:29.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj389 datafile 2021-07-26T06:17:29.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:29.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj390 datafile 2021-07-26T06:17:29.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:29.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj391 datafile 2021-07-26T06:17:29.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:29.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj392 datafile 2021-07-26T06:17:29.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:29.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj393 datafile 2021-07-26T06:17:30.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:30.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj394 datafile 2021-07-26T06:17:30.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:30.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj395 datafile 2021-07-26T06:17:30.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:30.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj396 datafile 2021-07-26T06:17:30.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:30.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj397 datafile 2021-07-26T06:17:30.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:30.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj398 datafile 2021-07-26T06:17:30.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:30.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj399 datafile 2021-07-26T06:17:30.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:30.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj400 datafile 2021-07-26T06:17:31.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:31.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj401 datafile 2021-07-26T06:17:31.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:31.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj402 datafile 2021-07-26T06:17:31.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:31.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj403 datafile 2021-07-26T06:17:31.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:31.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj404 datafile 2021-07-26T06:17:31.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:31.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj405 datafile 2021-07-26T06:17:31.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:31.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj406 datafile 2021-07-26T06:17:31.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:31.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj407 datafile 2021-07-26T06:17:31.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:31.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj408 datafile 2021-07-26T06:17:32.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:32.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj409 datafile 2021-07-26T06:17:32.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:32.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj410 datafile 2021-07-26T06:17:32.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:32.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj411 datafile 2021-07-26T06:17:32.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:32.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj412 datafile 2021-07-26T06:17:32.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:32.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj413 datafile 2021-07-26T06:17:32.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:32.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj414 datafile 2021-07-26T06:17:32.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:32.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj415 datafile 2021-07-26T06:17:33.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:33.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj416 datafile 2021-07-26T06:17:33.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:33.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj417 datafile 2021-07-26T06:17:33.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:33.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj418 datafile 2021-07-26T06:17:33.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:33.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj419 datafile 2021-07-26T06:17:33.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:33.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj420 datafile 2021-07-26T06:17:33.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:33.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj421 datafile 2021-07-26T06:17:33.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:33.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj422 datafile 2021-07-26T06:17:34.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:34.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj423 datafile 2021-07-26T06:17:34.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:34.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj424 datafile 2021-07-26T06:17:34.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:34.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj425 datafile 2021-07-26T06:17:34.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:34.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj426 datafile 2021-07-26T06:17:34.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:34.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj427 datafile 2021-07-26T06:17:34.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:34.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj428 datafile 2021-07-26T06:17:34.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:34.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj429 datafile 2021-07-26T06:17:35.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:35.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj430 datafile 2021-07-26T06:17:35.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:35.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj431 datafile 2021-07-26T06:17:35.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:35.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj432 datafile 2021-07-26T06:17:35.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:35.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj433 datafile 2021-07-26T06:17:35.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:35.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj434 datafile 2021-07-26T06:17:35.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:35.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj435 datafile 2021-07-26T06:17:35.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:35.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj436 datafile 2021-07-26T06:17:36.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:36.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj437 datafile 2021-07-26T06:17:36.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:36.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj438 datafile 2021-07-26T06:17:36.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:36.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj439 datafile 2021-07-26T06:17:36.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:36.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj440 datafile 2021-07-26T06:17:36.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:36.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj441 datafile 2021-07-26T06:17:36.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:36.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj442 datafile 2021-07-26T06:17:36.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:36.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj443 datafile 2021-07-26T06:17:37.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:37.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj444 datafile 2021-07-26T06:17:37.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:37.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj445 datafile 2021-07-26T06:17:37.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:37.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj446 datafile 2021-07-26T06:17:37.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:37.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj447 datafile 2021-07-26T06:17:37.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:37.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj448 datafile 2021-07-26T06:17:37.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:37.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj449 datafile 2021-07-26T06:17:38.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:38.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj450 datafile 2021-07-26T06:17:38.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:38.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj451 datafile 2021-07-26T06:17:38.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:38.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj452 datafile 2021-07-26T06:17:38.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:38.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj453 datafile 2021-07-26T06:17:38.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:38.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj454 datafile 2021-07-26T06:17:38.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:38.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj455 datafile 2021-07-26T06:17:38.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:38.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj456 datafile 2021-07-26T06:17:38.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:38.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj457 datafile 2021-07-26T06:17:39.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:39.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj458 datafile 2021-07-26T06:17:39.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:39.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj459 datafile 2021-07-26T06:17:39.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:39.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj460 datafile 2021-07-26T06:17:39.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:39.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj461 datafile 2021-07-26T06:17:39.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:39.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj462 datafile 2021-07-26T06:17:39.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:39.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj463 datafile 2021-07-26T06:17:39.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:39.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj464 datafile 2021-07-26T06:17:40.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:40.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj465 datafile 2021-07-26T06:17:40.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:40.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj466 datafile 2021-07-26T06:17:40.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:40.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj467 datafile 2021-07-26T06:17:40.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:40.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj468 datafile 2021-07-26T06:17:40.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:40.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj469 datafile 2021-07-26T06:17:40.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:40.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj470 datafile 2021-07-26T06:17:41.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:41.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj471 datafile 2021-07-26T06:17:41.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:41.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj472 datafile 2021-07-26T06:17:41.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:41.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj473 datafile 2021-07-26T06:17:41.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:41.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj474 datafile 2021-07-26T06:17:41.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:41.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj475 datafile 2021-07-26T06:17:41.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:41.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj476 datafile 2021-07-26T06:17:41.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:41.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj477 datafile 2021-07-26T06:17:42.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:42.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj478 datafile 2021-07-26T06:17:42.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:42.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj479 datafile 2021-07-26T06:17:42.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:42.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj480 datafile 2021-07-26T06:17:42.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:42.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj481 datafile 2021-07-26T06:17:42.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:42.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj482 datafile 2021-07-26T06:17:42.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:42.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj483 datafile 2021-07-26T06:17:42.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:42.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj484 datafile 2021-07-26T06:17:42.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:42.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj485 datafile 2021-07-26T06:17:43.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:43.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj486 datafile 2021-07-26T06:17:43.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:43.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj487 datafile 2021-07-26T06:17:43.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:43.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj488 datafile 2021-07-26T06:17:43.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:43.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj489 datafile 2021-07-26T06:17:43.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:43.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj490 datafile 2021-07-26T06:17:43.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:43.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj491 datafile 2021-07-26T06:17:43.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:43.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj492 datafile 2021-07-26T06:17:44.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:44.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj493 datafile 2021-07-26T06:17:44.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:44.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj494 datafile 2021-07-26T06:17:44.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:44.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj495 datafile 2021-07-26T06:17:44.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:44.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj496 datafile 2021-07-26T06:17:44.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:44.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj497 datafile 2021-07-26T06:17:44.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:44.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj498 datafile 2021-07-26T06:17:44.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:44.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj499 datafile 2021-07-26T06:17:45.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:45.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj500 datafile 2021-07-26T06:17:45.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:45.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj501 datafile 2021-07-26T06:17:45.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:45.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj502 datafile 2021-07-26T06:17:45.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:45.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj503 datafile 2021-07-26T06:17:45.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:45.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj504 datafile 2021-07-26T06:17:45.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:45.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj505 datafile 2021-07-26T06:17:46.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:46.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj506 datafile 2021-07-26T06:17:46.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:46.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj507 datafile 2021-07-26T06:17:46.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:46.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj508 datafile 2021-07-26T06:17:46.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:46.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj509 datafile 2021-07-26T06:17:46.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:46.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj510 datafile 2021-07-26T06:17:46.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:46.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj511 datafile 2021-07-26T06:17:46.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:46.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj512 datafile 2021-07-26T06:17:46.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:46.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj513 datafile 2021-07-26T06:17:47.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:47.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj514 datafile 2021-07-26T06:17:47.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:47.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj515 datafile 2021-07-26T06:17:47.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:47.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj516 datafile 2021-07-26T06:17:47.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:47.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj517 datafile 2021-07-26T06:17:47.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:47.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj518 datafile 2021-07-26T06:17:47.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:47.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj519 datafile 2021-07-26T06:17:47.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:48.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj520 datafile 2021-07-26T06:17:48.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:48.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj521 datafile 2021-07-26T06:17:48.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:48.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj522 datafile 2021-07-26T06:17:48.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:48.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj523 datafile 2021-07-26T06:17:48.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:48.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj524 datafile 2021-07-26T06:17:48.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:48.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj525 datafile 2021-07-26T06:17:48.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:48.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj526 datafile 2021-07-26T06:17:48.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:48.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj527 datafile 2021-07-26T06:17:49.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:49.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj528 datafile 2021-07-26T06:17:49.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:49.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj529 datafile 2021-07-26T06:17:49.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:49.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj530 datafile 2021-07-26T06:17:49.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:49.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj531 datafile 2021-07-26T06:17:49.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:49.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj532 datafile 2021-07-26T06:17:49.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:49.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj533 datafile 2021-07-26T06:17:49.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:49.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj534 datafile 2021-07-26T06:17:50.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:50.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj535 datafile 2021-07-26T06:17:50.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:50.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj536 datafile 2021-07-26T06:17:50.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:50.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj537 datafile 2021-07-26T06:17:50.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:50.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj538 datafile 2021-07-26T06:17:50.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:50.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj539 datafile 2021-07-26T06:17:51.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:51.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj540 datafile 2021-07-26T06:17:51.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:51.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj541 datafile 2021-07-26T06:17:51.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:51.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj542 datafile 2021-07-26T06:17:51.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:51.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj543 datafile 2021-07-26T06:17:51.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:51.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj544 datafile 2021-07-26T06:17:51.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:51.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj545 datafile 2021-07-26T06:17:51.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:51.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj546 datafile 2021-07-26T06:17:52.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:52.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj547 datafile 2021-07-26T06:17:52.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:52.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj548 datafile 2021-07-26T06:17:52.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:52.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj549 datafile 2021-07-26T06:17:52.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:52.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj550 datafile 2021-07-26T06:17:52.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:52.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj551 datafile 2021-07-26T06:17:52.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:52.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj552 datafile 2021-07-26T06:17:52.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:52.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj553 datafile 2021-07-26T06:17:53.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:53.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj554 datafile 2021-07-26T06:17:53.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:53.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj555 datafile 2021-07-26T06:17:53.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:53.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj556 datafile 2021-07-26T06:17:53.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:53.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj557 datafile 2021-07-26T06:17:53.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:53.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj558 datafile 2021-07-26T06:17:53.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:53.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj559 datafile 2021-07-26T06:17:53.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:53.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj560 datafile 2021-07-26T06:17:54.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:54.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj561 datafile 2021-07-26T06:17:54.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:54.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj562 datafile 2021-07-26T06:17:54.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:54.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj563 datafile 2021-07-26T06:17:54.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:54.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj564 datafile 2021-07-26T06:17:54.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:54.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj565 datafile 2021-07-26T06:17:54.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:54.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj566 datafile 2021-07-26T06:17:54.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:54.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj567 datafile 2021-07-26T06:17:55.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:55.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj568 datafile 2021-07-26T06:17:55.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:55.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj569 datafile 2021-07-26T06:17:55.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:55.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj570 datafile 2021-07-26T06:17:55.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:55.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj571 datafile 2021-07-26T06:17:55.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:55.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj572 datafile 2021-07-26T06:17:55.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:55.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj573 datafile 2021-07-26T06:17:55.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:55.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj574 datafile 2021-07-26T06:17:56.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:56.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj575 datafile 2021-07-26T06:17:56.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:56.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj576 datafile 2021-07-26T06:17:56.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:56.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj577 datafile 2021-07-26T06:17:56.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:56.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj578 datafile 2021-07-26T06:17:56.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:56.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj579 datafile 2021-07-26T06:17:56.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:56.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj580 datafile 2021-07-26T06:17:56.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:56.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj581 datafile 2021-07-26T06:17:57.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:57.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj582 datafile 2021-07-26T06:17:57.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:57.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj583 datafile 2021-07-26T06:17:57.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:57.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj584 datafile 2021-07-26T06:17:57.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:57.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj585 datafile 2021-07-26T06:17:57.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:57.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj586 datafile 2021-07-26T06:17:57.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:57.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj587 datafile 2021-07-26T06:17:57.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:57.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj588 datafile 2021-07-26T06:17:58.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:58.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj589 datafile 2021-07-26T06:17:58.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:58.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj590 datafile 2021-07-26T06:17:58.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:58.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj591 datafile 2021-07-26T06:17:58.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:58.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj592 datafile 2021-07-26T06:17:58.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:58.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj593 datafile 2021-07-26T06:17:58.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:58.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj594 datafile 2021-07-26T06:17:58.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:58.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj595 datafile 2021-07-26T06:17:59.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:59.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj596 datafile 2021-07-26T06:17:59.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:59.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj597 datafile 2021-07-26T06:17:59.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:59.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj598 datafile 2021-07-26T06:17:59.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:59.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj599 datafile 2021-07-26T06:17:59.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:59.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj600 datafile 2021-07-26T06:17:59.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:59.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj601 datafile 2021-07-26T06:17:59.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:17:59.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj602 datafile 2021-07-26T06:18:00.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:00.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj603 datafile 2021-07-26T06:18:00.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:00.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj604 datafile 2021-07-26T06:18:00.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:00.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj605 datafile 2021-07-26T06:18:00.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:00.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj606 datafile 2021-07-26T06:18:00.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:00.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj607 datafile 2021-07-26T06:18:00.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:00.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj608 datafile 2021-07-26T06:18:00.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:00.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj609 datafile 2021-07-26T06:18:01.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:01.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj610 datafile 2021-07-26T06:18:01.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:01.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj611 datafile 2021-07-26T06:18:01.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:01.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj612 datafile 2021-07-26T06:18:01.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:01.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj613 datafile 2021-07-26T06:18:01.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:01.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj614 datafile 2021-07-26T06:18:01.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:01.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj615 datafile 2021-07-26T06:18:02.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:02.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj616 datafile 2021-07-26T06:18:02.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:02.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj617 datafile 2021-07-26T06:18:02.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:02.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj618 datafile 2021-07-26T06:18:02.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:02.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj619 datafile 2021-07-26T06:18:02.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:02.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj620 datafile 2021-07-26T06:18:02.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:02.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj621 datafile 2021-07-26T06:18:02.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:02.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj622 datafile 2021-07-26T06:18:03.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:03.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj623 datafile 2021-07-26T06:18:03.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:03.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj624 datafile 2021-07-26T06:18:03.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:03.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj625 datafile 2021-07-26T06:18:03.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:03.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj626 datafile 2021-07-26T06:18:03.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:03.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj627 datafile 2021-07-26T06:18:03.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:03.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj628 datafile 2021-07-26T06:18:03.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:03.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj629 datafile 2021-07-26T06:18:04.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:04.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj630 datafile 2021-07-26T06:18:04.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:04.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj631 datafile 2021-07-26T06:18:04.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:04.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj632 datafile 2021-07-26T06:18:04.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:04.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj633 datafile 2021-07-26T06:18:04.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:04.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj634 datafile 2021-07-26T06:18:04.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:04.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj635 datafile 2021-07-26T06:18:04.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:04.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj636 datafile 2021-07-26T06:18:04.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:04.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj637 datafile 2021-07-26T06:18:05.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:05.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj638 datafile 2021-07-26T06:18:05.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:05.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj639 datafile 2021-07-26T06:18:05.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:05.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj640 datafile 2021-07-26T06:18:05.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:05.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj641 datafile 2021-07-26T06:18:05.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:05.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj642 datafile 2021-07-26T06:18:05.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:05.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj643 datafile 2021-07-26T06:18:05.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:05.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj644 datafile 2021-07-26T06:18:06.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:06.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj645 datafile 2021-07-26T06:18:06.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:06.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj646 datafile 2021-07-26T06:18:06.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:06.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj647 datafile 2021-07-26T06:18:06.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:06.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj648 datafile 2021-07-26T06:18:06.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:06.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj649 datafile 2021-07-26T06:18:06.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:06.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj650 datafile 2021-07-26T06:18:07.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:07.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj651 datafile 2021-07-26T06:18:07.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:07.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj652 datafile 2021-07-26T06:18:07.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:07.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj653 datafile 2021-07-26T06:18:07.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:07.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj654 datafile 2021-07-26T06:18:07.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:07.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj655 datafile 2021-07-26T06:18:07.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:07.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj656 datafile 2021-07-26T06:18:07.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:07.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj657 datafile 2021-07-26T06:18:07.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:07.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj658 datafile 2021-07-26T06:18:08.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:08.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj659 datafile 2021-07-26T06:18:08.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:08.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj660 datafile 2021-07-26T06:18:08.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:08.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj661 datafile 2021-07-26T06:18:08.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:08.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj662 datafile 2021-07-26T06:18:08.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:08.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj663 datafile 2021-07-26T06:18:08.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:08.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj664 datafile 2021-07-26T06:18:08.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:08.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj665 datafile 2021-07-26T06:18:09.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:09.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj666 datafile 2021-07-26T06:18:09.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:09.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj667 datafile 2021-07-26T06:18:09.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:09.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj668 datafile 2021-07-26T06:18:09.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:09.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj669 datafile 2021-07-26T06:18:09.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:09.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj670 datafile 2021-07-26T06:18:09.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:09.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj671 datafile 2021-07-26T06:18:09.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:09.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj672 datafile 2021-07-26T06:18:10.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:10.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj673 datafile 2021-07-26T06:18:10.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:10.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj674 datafile 2021-07-26T06:18:10.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:10.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj675 datafile 2021-07-26T06:18:10.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:10.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj676 datafile 2021-07-26T06:18:10.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:10.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj677 datafile 2021-07-26T06:18:10.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:10.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj678 datafile 2021-07-26T06:18:10.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:10.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj679 datafile 2021-07-26T06:18:10.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:10.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj680 datafile 2021-07-26T06:18:11.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:11.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj681 datafile 2021-07-26T06:18:11.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:11.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj682 datafile 2021-07-26T06:18:11.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:11.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj683 datafile 2021-07-26T06:18:11.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:11.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj684 datafile 2021-07-26T06:18:11.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:11.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj685 datafile 2021-07-26T06:18:11.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:11.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj686 datafile 2021-07-26T06:18:11.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:11.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj687 datafile 2021-07-26T06:18:12.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:12.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj688 datafile 2021-07-26T06:18:12.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:12.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj689 datafile 2021-07-26T06:18:12.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:12.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj690 datafile 2021-07-26T06:18:12.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:12.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj691 datafile 2021-07-26T06:18:12.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:12.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj692 datafile 2021-07-26T06:18:12.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:12.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj693 datafile 2021-07-26T06:18:12.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:12.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj694 datafile 2021-07-26T06:18:12.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:12.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj695 datafile 2021-07-26T06:18:13.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:13.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj696 datafile 2021-07-26T06:18:13.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:13.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj697 datafile 2021-07-26T06:18:13.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:13.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj698 datafile 2021-07-26T06:18:13.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:13.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj699 datafile 2021-07-26T06:18:13.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:13.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj700 datafile 2021-07-26T06:18:13.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:13.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj701 datafile 2021-07-26T06:18:13.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:13.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj702 datafile 2021-07-26T06:18:14.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:14.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj703 datafile 2021-07-26T06:18:14.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:14.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj704 datafile 2021-07-26T06:18:14.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:14.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj705 datafile 2021-07-26T06:18:14.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:14.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj706 datafile 2021-07-26T06:18:14.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:14.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj707 datafile 2021-07-26T06:18:14.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:14.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj708 datafile 2021-07-26T06:18:14.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:14.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj709 datafile 2021-07-26T06:18:15.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:15.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj710 datafile 2021-07-26T06:18:15.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:15.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj711 datafile 2021-07-26T06:18:15.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:15.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj712 datafile 2021-07-26T06:18:15.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:15.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj713 datafile 2021-07-26T06:18:15.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:15.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj714 datafile 2021-07-26T06:18:15.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:15.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj715 datafile 2021-07-26T06:18:15.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:15.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj716 datafile 2021-07-26T06:18:16.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:16.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj717 datafile 2021-07-26T06:18:16.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:16.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj718 datafile 2021-07-26T06:18:16.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:16.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj719 datafile 2021-07-26T06:18:16.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:16.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj720 datafile 2021-07-26T06:18:16.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:16.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj721 datafile 2021-07-26T06:18:16.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:16.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj722 datafile 2021-07-26T06:18:16.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:16.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj723 datafile 2021-07-26T06:18:16.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:16.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj724 datafile 2021-07-26T06:18:17.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:17.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj725 datafile 2021-07-26T06:18:17.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:17.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj726 datafile 2021-07-26T06:18:17.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:17.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj727 datafile 2021-07-26T06:18:17.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:17.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj728 datafile 2021-07-26T06:18:17.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:17.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj729 datafile 2021-07-26T06:18:17.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:17.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj730 datafile 2021-07-26T06:18:18.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:18.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj731 datafile 2021-07-26T06:18:18.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:18.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj732 datafile 2021-07-26T06:18:18.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:18.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj733 datafile 2021-07-26T06:18:18.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:18.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj734 datafile 2021-07-26T06:18:18.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:18.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj735 datafile 2021-07-26T06:18:18.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:18.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj736 datafile 2021-07-26T06:18:18.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:18.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj737 datafile 2021-07-26T06:18:19.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:19.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj738 datafile 2021-07-26T06:18:19.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:19.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj739 datafile 2021-07-26T06:18:19.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:19.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj740 datafile 2021-07-26T06:18:19.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:19.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj741 datafile 2021-07-26T06:18:19.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:19.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj742 datafile 2021-07-26T06:18:19.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:19.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj743 datafile 2021-07-26T06:18:19.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:19.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj744 datafile 2021-07-26T06:18:20.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:20.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj745 datafile 2021-07-26T06:18:20.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:20.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj746 datafile 2021-07-26T06:18:20.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:20.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj747 datafile 2021-07-26T06:18:20.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:20.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj748 datafile 2021-07-26T06:18:20.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:20.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj749 datafile 2021-07-26T06:18:20.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:20.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj750 datafile 2021-07-26T06:18:20.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:20.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj751 datafile 2021-07-26T06:18:21.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:21.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj752 datafile 2021-07-26T06:18:21.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:21.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj753 datafile 2021-07-26T06:18:21.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:21.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj754 datafile 2021-07-26T06:18:21.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:21.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj755 datafile 2021-07-26T06:18:21.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:21.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj756 datafile 2021-07-26T06:18:22.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:22.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj757 datafile 2021-07-26T06:18:22.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:22.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj758 datafile 2021-07-26T06:18:22.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:22.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj759 datafile 2021-07-26T06:18:22.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:22.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj760 datafile 2021-07-26T06:18:22.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:22.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj761 datafile 2021-07-26T06:18:22.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:22.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj762 datafile 2021-07-26T06:18:22.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:22.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj763 datafile 2021-07-26T06:18:23.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:23.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj764 datafile 2021-07-26T06:18:23.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:23.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj765 datafile 2021-07-26T06:18:23.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:23.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj766 datafile 2021-07-26T06:18:23.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:23.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj767 datafile 2021-07-26T06:18:23.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:23.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj768 datafile 2021-07-26T06:18:23.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:23.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj769 datafile 2021-07-26T06:18:23.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:23.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj770 datafile 2021-07-26T06:18:24.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:24.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj771 datafile 2021-07-26T06:18:24.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:24.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj772 datafile 2021-07-26T06:18:24.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:24.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj773 datafile 2021-07-26T06:18:24.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:24.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj774 datafile 2021-07-26T06:18:24.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:24.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj775 datafile 2021-07-26T06:18:24.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:24.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj776 datafile 2021-07-26T06:18:24.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:24.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj777 datafile 2021-07-26T06:18:25.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:25.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj778 datafile 2021-07-26T06:18:25.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:25.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj779 datafile 2021-07-26T06:18:25.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:25.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj780 datafile 2021-07-26T06:18:25.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:25.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj781 datafile 2021-07-26T06:18:25.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:25.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj782 datafile 2021-07-26T06:18:25.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:25.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj783 datafile 2021-07-26T06:18:25.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:25.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj784 datafile 2021-07-26T06:18:26.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:26.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj785 datafile 2021-07-26T06:18:26.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:26.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj786 datafile 2021-07-26T06:18:26.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:26.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj787 datafile 2021-07-26T06:18:26.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:26.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj788 datafile 2021-07-26T06:18:26.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:26.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj789 datafile 2021-07-26T06:18:26.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:26.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj790 datafile 2021-07-26T06:18:26.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:26.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj791 datafile 2021-07-26T06:18:27.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:27.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj792 datafile 2021-07-26T06:18:27.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:27.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj793 datafile 2021-07-26T06:18:27.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:27.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj794 datafile 2021-07-26T06:18:27.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:27.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj795 datafile 2021-07-26T06:18:27.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:27.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj796 datafile 2021-07-26T06:18:27.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:27.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj797 datafile 2021-07-26T06:18:27.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:27.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj798 datafile 2021-07-26T06:18:28.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:28.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj799 datafile 2021-07-26T06:18:28.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:28.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj800 datafile 2021-07-26T06:18:28.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:28.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj801 datafile 2021-07-26T06:18:28.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:28.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj802 datafile 2021-07-26T06:18:28.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:28.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj803 datafile 2021-07-26T06:18:28.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:28.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj804 datafile 2021-07-26T06:18:28.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:28.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj805 datafile 2021-07-26T06:18:29.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:29.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj806 datafile 2021-07-26T06:18:29.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:29.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj807 datafile 2021-07-26T06:18:29.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:29.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj808 datafile 2021-07-26T06:18:29.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:29.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj809 datafile 2021-07-26T06:18:29.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:29.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj810 datafile 2021-07-26T06:18:29.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:29.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj811 datafile 2021-07-26T06:18:29.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:29.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj812 datafile 2021-07-26T06:18:30.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:30.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj813 datafile 2021-07-26T06:18:30.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:30.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj814 datafile 2021-07-26T06:18:30.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:30.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj815 datafile 2021-07-26T06:18:30.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:30.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj816 datafile 2021-07-26T06:18:30.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:30.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj817 datafile 2021-07-26T06:18:30.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:30.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj818 datafile 2021-07-26T06:18:30.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:30.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj819 datafile 2021-07-26T06:18:31.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:31.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj820 datafile 2021-07-26T06:18:31.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:31.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj821 datafile 2021-07-26T06:18:31.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:31.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj822 datafile 2021-07-26T06:18:31.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:31.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj823 datafile 2021-07-26T06:18:31.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:31.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj824 datafile 2021-07-26T06:18:31.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:31.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj825 datafile 2021-07-26T06:18:31.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:31.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj826 datafile 2021-07-26T06:18:32.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:32.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj827 datafile 2021-07-26T06:18:32.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:32.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj828 datafile 2021-07-26T06:18:32.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:32.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj829 datafile 2021-07-26T06:18:32.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:32.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj830 datafile 2021-07-26T06:18:32.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:32.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj831 datafile 2021-07-26T06:18:32.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:32.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj832 datafile 2021-07-26T06:18:32.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:32.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj833 datafile 2021-07-26T06:18:33.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:33.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj834 datafile 2021-07-26T06:18:33.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:33.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj835 datafile 2021-07-26T06:18:33.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:33.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj836 datafile 2021-07-26T06:18:33.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:33.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj837 datafile 2021-07-26T06:18:33.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:33.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj838 datafile 2021-07-26T06:18:33.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:33.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj839 datafile 2021-07-26T06:18:34.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:34.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj840 datafile 2021-07-26T06:18:34.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:34.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj841 datafile 2021-07-26T06:18:34.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:34.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj842 datafile 2021-07-26T06:18:34.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:34.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj843 datafile 2021-07-26T06:18:34.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:34.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj844 datafile 2021-07-26T06:18:34.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:34.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj845 datafile 2021-07-26T06:18:34.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:34.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj846 datafile 2021-07-26T06:18:35.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:35.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj847 datafile 2021-07-26T06:18:35.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:35.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj848 datafile 2021-07-26T06:18:35.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:35.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj849 datafile 2021-07-26T06:18:35.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:35.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj850 datafile 2021-07-26T06:18:35.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:35.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj851 datafile 2021-07-26T06:18:35.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:35.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj852 datafile 2021-07-26T06:18:35.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:35.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj853 datafile 2021-07-26T06:18:36.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:36.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj854 datafile 2021-07-26T06:18:36.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:36.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj855 datafile 2021-07-26T06:18:36.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:36.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj856 datafile 2021-07-26T06:18:36.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:36.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj857 datafile 2021-07-26T06:18:36.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:36.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj858 datafile 2021-07-26T06:18:36.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:36.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj859 datafile 2021-07-26T06:18:36.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:36.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj860 datafile 2021-07-26T06:18:37.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:37.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj861 datafile 2021-07-26T06:18:37.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:37.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj862 datafile 2021-07-26T06:18:37.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:37.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj863 datafile 2021-07-26T06:18:37.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:37.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj864 datafile 2021-07-26T06:18:37.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:37.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj865 datafile 2021-07-26T06:18:37.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:37.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj866 datafile 2021-07-26T06:18:37.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:37.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj867 datafile 2021-07-26T06:18:37.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:37.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj868 datafile 2021-07-26T06:18:38.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:38.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj869 datafile 2021-07-26T06:18:38.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:38.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj870 datafile 2021-07-26T06:18:38.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:38.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj871 datafile 2021-07-26T06:18:38.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:38.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj872 datafile 2021-07-26T06:18:38.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:38.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj873 datafile 2021-07-26T06:18:38.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:38.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj874 datafile 2021-07-26T06:18:38.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:38.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj875 datafile 2021-07-26T06:18:39.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:39.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj876 datafile 2021-07-26T06:18:39.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:39.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj877 datafile 2021-07-26T06:18:39.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:39.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj878 datafile 2021-07-26T06:18:39.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:39.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj879 datafile 2021-07-26T06:18:39.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:39.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj880 datafile 2021-07-26T06:18:39.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:39.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj881 datafile 2021-07-26T06:18:39.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:39.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj882 datafile 2021-07-26T06:18:40.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:40.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj883 datafile 2021-07-26T06:18:40.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:40.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj884 datafile 2021-07-26T06:18:40.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:40.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj885 datafile 2021-07-26T06:18:40.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:40.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj886 datafile 2021-07-26T06:18:40.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:40.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj887 datafile 2021-07-26T06:18:40.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:40.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj888 datafile 2021-07-26T06:18:40.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:40.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj889 datafile 2021-07-26T06:18:41.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:41.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj890 datafile 2021-07-26T06:18:41.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:41.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj891 datafile 2021-07-26T06:18:41.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:41.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj892 datafile 2021-07-26T06:18:41.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:41.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj893 datafile 2021-07-26T06:18:41.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:41.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj894 datafile 2021-07-26T06:18:41.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:41.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj895 datafile 2021-07-26T06:18:41.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:41.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj896 datafile 2021-07-26T06:18:42.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:42.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj897 datafile 2021-07-26T06:18:42.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:42.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj898 datafile 2021-07-26T06:18:42.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:42.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj899 datafile 2021-07-26T06:18:42.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:42.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj900 datafile 2021-07-26T06:18:42.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:42.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj901 datafile 2021-07-26T06:18:42.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:42.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj902 datafile 2021-07-26T06:18:42.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:42.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj903 datafile 2021-07-26T06:18:43.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:43.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj904 datafile 2021-07-26T06:18:43.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:43.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj905 datafile 2021-07-26T06:18:43.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:43.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj906 datafile 2021-07-26T06:18:43.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:43.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj907 datafile 2021-07-26T06:18:43.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:43.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj908 datafile 2021-07-26T06:18:43.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:43.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj909 datafile 2021-07-26T06:18:43.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:43.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj910 datafile 2021-07-26T06:18:44.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:44.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj911 datafile 2021-07-26T06:18:44.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:44.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj912 datafile 2021-07-26T06:18:44.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:44.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj913 datafile 2021-07-26T06:18:44.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:44.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj914 datafile 2021-07-26T06:18:44.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:44.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj915 datafile 2021-07-26T06:18:44.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:44.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj916 datafile 2021-07-26T06:18:44.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:44.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj917 datafile 2021-07-26T06:18:45.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:45.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj918 datafile 2021-07-26T06:18:45.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:45.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj919 datafile 2021-07-26T06:18:45.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:45.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj920 datafile 2021-07-26T06:18:45.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:45.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj921 datafile 2021-07-26T06:18:45.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:45.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj922 datafile 2021-07-26T06:18:45.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:45.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj923 datafile 2021-07-26T06:18:45.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:45.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj924 datafile 2021-07-26T06:18:46.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:46.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj925 datafile 2021-07-26T06:18:46.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:46.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj926 datafile 2021-07-26T06:18:46.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:46.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj927 datafile 2021-07-26T06:18:46.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:46.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj928 datafile 2021-07-26T06:18:46.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:46.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj929 datafile 2021-07-26T06:18:46.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:46.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj930 datafile 2021-07-26T06:18:46.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:46.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj931 datafile 2021-07-26T06:18:47.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:47.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj932 datafile 2021-07-26T06:18:47.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:47.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj933 datafile 2021-07-26T06:18:47.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:47.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj934 datafile 2021-07-26T06:18:47.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:47.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj935 datafile 2021-07-26T06:18:47.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:47.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj936 datafile 2021-07-26T06:18:47.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:47.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj937 datafile 2021-07-26T06:18:47.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:47.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj938 datafile 2021-07-26T06:18:48.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:48.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj939 datafile 2021-07-26T06:18:48.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:48.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj940 datafile 2021-07-26T06:18:48.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:48.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj941 datafile 2021-07-26T06:18:48.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:48.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj942 datafile 2021-07-26T06:18:48.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:48.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj943 datafile 2021-07-26T06:18:48.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:48.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj944 datafile 2021-07-26T06:18:48.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:48.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj945 datafile 2021-07-26T06:18:49.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:49.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj946 datafile 2021-07-26T06:18:49.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:49.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj947 datafile 2021-07-26T06:18:49.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:49.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj948 datafile 2021-07-26T06:18:49.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:49.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj949 datafile 2021-07-26T06:18:49.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:49.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj950 datafile 2021-07-26T06:18:49.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:49.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj951 datafile 2021-07-26T06:18:49.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:49.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj952 datafile 2021-07-26T06:18:50.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:50.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj953 datafile 2021-07-26T06:18:50.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:50.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj954 datafile 2021-07-26T06:18:50.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:50.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj955 datafile 2021-07-26T06:18:50.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:50.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj956 datafile 2021-07-26T06:18:50.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:50.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj957 datafile 2021-07-26T06:18:50.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:50.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj958 datafile 2021-07-26T06:18:50.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:50.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj959 datafile 2021-07-26T06:18:51.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:51.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj960 datafile 2021-07-26T06:18:51.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:51.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj961 datafile 2021-07-26T06:18:51.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:51.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj962 datafile 2021-07-26T06:18:51.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:51.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj963 datafile 2021-07-26T06:18:51.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:51.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj964 datafile 2021-07-26T06:18:51.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:51.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj965 datafile 2021-07-26T06:18:51.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:51.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj966 datafile 2021-07-26T06:18:52.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:52.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj967 datafile 2021-07-26T06:18:52.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:52.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj968 datafile 2021-07-26T06:18:52.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:52.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj969 datafile 2021-07-26T06:18:52.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:52.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj970 datafile 2021-07-26T06:18:52.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:52.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj971 datafile 2021-07-26T06:18:52.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:52.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj972 datafile 2021-07-26T06:18:53.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:53.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj973 datafile 2021-07-26T06:18:53.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:53.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj974 datafile 2021-07-26T06:18:53.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:53.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj975 datafile 2021-07-26T06:18:53.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:53.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj976 datafile 2021-07-26T06:18:53.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:53.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj977 datafile 2021-07-26T06:18:53.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:53.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj978 datafile 2021-07-26T06:18:54.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:54.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj979 datafile 2021-07-26T06:18:54.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:54.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj980 datafile 2021-07-26T06:18:54.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:54.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj981 datafile 2021-07-26T06:18:54.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:54.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj982 datafile 2021-07-26T06:18:54.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:54.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj983 datafile 2021-07-26T06:18:54.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:54.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj984 datafile 2021-07-26T06:18:54.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:54.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj985 datafile 2021-07-26T06:18:55.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:55.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj986 datafile 2021-07-26T06:18:55.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:55.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj987 datafile 2021-07-26T06:18:55.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:55.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj988 datafile 2021-07-26T06:18:55.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:55.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj989 datafile 2021-07-26T06:18:55.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:55.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj990 datafile 2021-07-26T06:18:55.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:55.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj991 datafile 2021-07-26T06:18:55.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:55.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj992 datafile 2021-07-26T06:18:56.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:56.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj993 datafile 2021-07-26T06:18:56.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:56.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj994 datafile 2021-07-26T06:18:56.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:56.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj995 datafile 2021-07-26T06:18:56.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:56.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj996 datafile 2021-07-26T06:18:56.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:56.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj997 datafile 2021-07-26T06:18:56.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:56.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj998 datafile 2021-07-26T06:18:56.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:56.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj999 datafile 2021-07-26T06:18:57.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:57.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj1000 datafile 2021-07-26T06:18:57.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:72: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2021-07-26T06:18:57.128 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: seq 1 1000 2021-07-26T06:18:57.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:57.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj1 datafile 2021-07-26T06:18:57.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:57.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj2 datafile 2021-07-26T06:18:57.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:57.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj3 datafile 2021-07-26T06:18:57.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:57.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj4 datafile 2021-07-26T06:18:57.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:57.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj5 datafile 2021-07-26T06:18:57.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:57.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj6 datafile 2021-07-26T06:18:57.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:57.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj7 datafile 2021-07-26T06:18:58.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:58.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj8 datafile 2021-07-26T06:18:58.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:58.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj9 datafile 2021-07-26T06:18:58.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:58.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj10 datafile 2021-07-26T06:18:58.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:58.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj11 datafile 2021-07-26T06:18:58.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:58.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj12 datafile 2021-07-26T06:18:58.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:58.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj13 datafile 2021-07-26T06:18:59.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:59.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj14 datafile 2021-07-26T06:18:59.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:59.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj15 datafile 2021-07-26T06:18:59.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:59.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj16 datafile 2021-07-26T06:18:59.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:59.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj17 datafile 2021-07-26T06:18:59.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:59.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj18 datafile 2021-07-26T06:18:59.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:59.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj19 datafile 2021-07-26T06:18:59.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:18:59.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj20 datafile 2021-07-26T06:19:00.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:00.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj21 datafile 2021-07-26T06:19:00.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:00.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj22 datafile 2021-07-26T06:19:00.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:00.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj23 datafile 2021-07-26T06:19:00.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:00.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj24 datafile 2021-07-26T06:19:00.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:00.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj25 datafile 2021-07-26T06:19:00.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:00.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj26 datafile 2021-07-26T06:19:00.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:00.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj27 datafile 2021-07-26T06:19:00.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:00.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj28 datafile 2021-07-26T06:19:01.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:01.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj29 datafile 2021-07-26T06:19:01.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:01.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj30 datafile 2021-07-26T06:19:01.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:01.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj31 datafile 2021-07-26T06:19:01.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:01.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj32 datafile 2021-07-26T06:19:01.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:01.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj33 datafile 2021-07-26T06:19:01.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:01.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj34 datafile 2021-07-26T06:19:01.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:01.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj35 datafile 2021-07-26T06:19:02.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:02.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj36 datafile 2021-07-26T06:19:02.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:02.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj37 datafile 2021-07-26T06:19:02.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:02.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj38 datafile 2021-07-26T06:19:02.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:02.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj39 datafile 2021-07-26T06:19:02.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:02.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj40 datafile 2021-07-26T06:19:02.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:02.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj41 datafile 2021-07-26T06:19:02.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:02.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj42 datafile 2021-07-26T06:19:03.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:03.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj43 datafile 2021-07-26T06:19:03.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:03.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj44 datafile 2021-07-26T06:19:03.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:03.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj45 datafile 2021-07-26T06:19:03.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:03.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj46 datafile 2021-07-26T06:19:03.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:03.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj47 datafile 2021-07-26T06:19:03.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:03.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj48 datafile 2021-07-26T06:19:04.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:04.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj49 datafile 2021-07-26T06:19:04.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:04.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj50 datafile 2021-07-26T06:19:04.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:04.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj51 datafile 2021-07-26T06:19:04.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:04.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj52 datafile 2021-07-26T06:19:04.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:04.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj53 datafile 2021-07-26T06:19:04.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:04.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj54 datafile 2021-07-26T06:19:04.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:04.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj55 datafile 2021-07-26T06:19:04.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:04.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj56 datafile 2021-07-26T06:19:05.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:05.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj57 datafile 2021-07-26T06:19:05.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:05.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj58 datafile 2021-07-26T06:19:05.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:05.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj59 datafile 2021-07-26T06:19:05.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:05.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj60 datafile 2021-07-26T06:19:05.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:05.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj61 datafile 2021-07-26T06:19:05.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:05.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj62 datafile 2021-07-26T06:19:05.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:05.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj63 datafile 2021-07-26T06:19:06.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:06.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj64 datafile 2021-07-26T06:19:06.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:06.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj65 datafile 2021-07-26T06:19:06.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:06.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj66 datafile 2021-07-26T06:19:06.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:06.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj67 datafile 2021-07-26T06:19:06.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:06.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj68 datafile 2021-07-26T06:19:06.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:06.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj69 datafile 2021-07-26T06:19:06.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:06.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj70 datafile 2021-07-26T06:19:07.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:07.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj71 datafile 2021-07-26T06:19:07.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:07.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj72 datafile 2021-07-26T06:19:07.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:07.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj73 datafile 2021-07-26T06:19:07.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:07.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj74 datafile 2021-07-26T06:19:07.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:07.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj75 datafile 2021-07-26T06:19:07.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:07.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj76 datafile 2021-07-26T06:19:07.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:07.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj77 datafile 2021-07-26T06:19:07.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:07.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj78 datafile 2021-07-26T06:19:08.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:08.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj79 datafile 2021-07-26T06:19:08.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:08.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj80 datafile 2021-07-26T06:19:08.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:08.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj81 datafile 2021-07-26T06:19:08.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:08.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj82 datafile 2021-07-26T06:19:08.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:08.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj83 datafile 2021-07-26T06:19:08.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:08.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj84 datafile 2021-07-26T06:19:08.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:08.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj85 datafile 2021-07-26T06:19:09.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:09.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj86 datafile 2021-07-26T06:19:09.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:09.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj87 datafile 2021-07-26T06:19:09.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:09.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj88 datafile 2021-07-26T06:19:09.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:09.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj89 datafile 2021-07-26T06:19:09.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:09.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj90 datafile 2021-07-26T06:19:09.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:09.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj91 datafile 2021-07-26T06:19:09.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:09.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj92 datafile 2021-07-26T06:19:10.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:10.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj93 datafile 2021-07-26T06:19:10.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:10.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj94 datafile 2021-07-26T06:19:10.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:10.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj95 datafile 2021-07-26T06:19:10.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:10.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj96 datafile 2021-07-26T06:19:10.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:10.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj97 datafile 2021-07-26T06:19:10.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:10.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj98 datafile 2021-07-26T06:19:10.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:10.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj99 datafile 2021-07-26T06:19:10.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:10.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj100 datafile 2021-07-26T06:19:11.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:11.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj101 datafile 2021-07-26T06:19:11.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:11.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj102 datafile 2021-07-26T06:19:11.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:11.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj103 datafile 2021-07-26T06:19:11.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:11.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj104 datafile 2021-07-26T06:19:11.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:11.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj105 datafile 2021-07-26T06:19:11.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:11.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj106 datafile 2021-07-26T06:19:11.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:11.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj107 datafile 2021-07-26T06:19:12.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:12.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj108 datafile 2021-07-26T06:19:12.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:12.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj109 datafile 2021-07-26T06:19:12.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:12.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj110 datafile 2021-07-26T06:19:12.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:12.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj111 datafile 2021-07-26T06:19:12.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:12.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj112 datafile 2021-07-26T06:19:12.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:12.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj113 datafile 2021-07-26T06:19:12.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:12.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj114 datafile 2021-07-26T06:19:12.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:12.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj115 datafile 2021-07-26T06:19:13.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:13.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj116 datafile 2021-07-26T06:19:13.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:13.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj117 datafile 2021-07-26T06:19:13.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:13.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj118 datafile 2021-07-26T06:19:13.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:13.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj119 datafile 2021-07-26T06:19:13.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:13.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj120 datafile 2021-07-26T06:19:13.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:13.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj121 datafile 2021-07-26T06:19:14.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:14.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj122 datafile 2021-07-26T06:19:14.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:14.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj123 datafile 2021-07-26T06:19:14.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:14.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj124 datafile 2021-07-26T06:19:14.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:14.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj125 datafile 2021-07-26T06:19:14.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:14.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj126 datafile 2021-07-26T06:19:14.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:14.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj127 datafile 2021-07-26T06:19:14.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:14.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj128 datafile 2021-07-26T06:19:14.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:15.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj129 datafile 2021-07-26T06:19:15.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:15.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj130 datafile 2021-07-26T06:19:15.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:15.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj131 datafile 2021-07-26T06:19:15.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:15.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj132 datafile 2021-07-26T06:19:15.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:15.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj133 datafile 2021-07-26T06:19:15.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:15.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj134 datafile 2021-07-26T06:19:15.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:15.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj135 datafile 2021-07-26T06:19:16.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:16.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj136 datafile 2021-07-26T06:19:16.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:16.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj137 datafile 2021-07-26T06:19:16.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:16.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj138 datafile 2021-07-26T06:19:16.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:16.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj139 datafile 2021-07-26T06:19:16.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:16.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj140 datafile 2021-07-26T06:19:16.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:16.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj141 datafile 2021-07-26T06:19:16.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:16.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj142 datafile 2021-07-26T06:19:16.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:16.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj143 datafile 2021-07-26T06:19:17.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:17.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj144 datafile 2021-07-26T06:19:17.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:17.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj145 datafile 2021-07-26T06:19:17.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:17.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj146 datafile 2021-07-26T06:19:17.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:17.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj147 datafile 2021-07-26T06:19:17.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:17.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj148 datafile 2021-07-26T06:19:17.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:17.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj149 datafile 2021-07-26T06:19:17.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:17.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj150 datafile 2021-07-26T06:19:18.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:18.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj151 datafile 2021-07-26T06:19:18.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:18.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj152 datafile 2021-07-26T06:19:18.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:18.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj153 datafile 2021-07-26T06:19:18.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:18.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj154 datafile 2021-07-26T06:19:18.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:18.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj155 datafile 2021-07-26T06:19:18.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:18.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj156 datafile 2021-07-26T06:19:18.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:18.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj157 datafile 2021-07-26T06:19:19.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:19.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj158 datafile 2021-07-26T06:19:19.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:19.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj159 datafile 2021-07-26T06:19:19.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:19.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj160 datafile 2021-07-26T06:19:19.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:19.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj161 datafile 2021-07-26T06:19:19.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:19.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj162 datafile 2021-07-26T06:19:19.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:19.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj163 datafile 2021-07-26T06:19:19.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:19.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj164 datafile 2021-07-26T06:19:20.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:20.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj165 datafile 2021-07-26T06:19:20.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:20.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj166 datafile 2021-07-26T06:19:20.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:20.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj167 datafile 2021-07-26T06:19:20.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:20.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj168 datafile 2021-07-26T06:19:20.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:20.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj169 datafile 2021-07-26T06:19:20.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:20.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj170 datafile 2021-07-26T06:19:20.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:20.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj171 datafile 2021-07-26T06:19:21.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:21.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj172 datafile 2021-07-26T06:19:21.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:21.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj173 datafile 2021-07-26T06:19:21.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:21.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj174 datafile 2021-07-26T06:19:21.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:21.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj175 datafile 2021-07-26T06:19:21.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:21.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj176 datafile 2021-07-26T06:19:21.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:21.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj177 datafile 2021-07-26T06:19:21.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:21.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj178 datafile 2021-07-26T06:19:22.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:22.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj179 datafile 2021-07-26T06:19:22.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:22.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj180 datafile 2021-07-26T06:19:22.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:22.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj181 datafile 2021-07-26T06:19:22.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:22.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj182 datafile 2021-07-26T06:19:22.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:22.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj183 datafile 2021-07-26T06:19:22.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:22.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj184 datafile 2021-07-26T06:19:22.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:22.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj185 datafile 2021-07-26T06:19:22.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:22.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj186 datafile 2021-07-26T06:19:23.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:23.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj187 datafile 2021-07-26T06:19:23.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:23.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj188 datafile 2021-07-26T06:19:23.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:23.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj189 datafile 2021-07-26T06:19:23.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:23.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj190 datafile 2021-07-26T06:19:23.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:23.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj191 datafile 2021-07-26T06:19:23.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:23.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj192 datafile 2021-07-26T06:19:24.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:24.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj193 datafile 2021-07-26T06:19:24.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:24.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj194 datafile 2021-07-26T06:19:24.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:24.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj195 datafile 2021-07-26T06:19:24.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:24.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj196 datafile 2021-07-26T06:19:24.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:24.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj197 datafile 2021-07-26T06:19:25.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:25.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj198 datafile 2021-07-26T06:19:25.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:25.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj199 datafile 2021-07-26T06:19:25.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:25.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj200 datafile 2021-07-26T06:19:25.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:25.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj201 datafile 2021-07-26T06:19:25.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:25.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj202 datafile 2021-07-26T06:19:25.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:25.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj203 datafile 2021-07-26T06:19:25.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:25.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj204 datafile 2021-07-26T06:19:26.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:26.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj205 datafile 2021-07-26T06:19:26.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:26.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj206 datafile 2021-07-26T06:19:26.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:26.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj207 datafile 2021-07-26T06:19:26.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:26.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj208 datafile 2021-07-26T06:19:26.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:26.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj209 datafile 2021-07-26T06:19:26.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:26.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj210 datafile 2021-07-26T06:19:26.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:26.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj211 datafile 2021-07-26T06:19:26.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:26.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj212 datafile 2021-07-26T06:19:27.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:27.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj213 datafile 2021-07-26T06:19:27.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:27.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj214 datafile 2021-07-26T06:19:27.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:27.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj215 datafile 2021-07-26T06:19:27.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:27.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj216 datafile 2021-07-26T06:19:27.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:27.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj217 datafile 2021-07-26T06:19:27.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:27.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj218 datafile 2021-07-26T06:19:27.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:27.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj219 datafile 2021-07-26T06:19:28.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:28.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj220 datafile 2021-07-26T06:19:28.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:28.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj221 datafile 2021-07-26T06:19:28.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:28.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj222 datafile 2021-07-26T06:19:28.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:28.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj223 datafile 2021-07-26T06:19:28.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:28.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj224 datafile 2021-07-26T06:19:28.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:28.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj225 datafile 2021-07-26T06:19:28.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:28.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj226 datafile 2021-07-26T06:19:29.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:29.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj227 datafile 2021-07-26T06:19:29.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:29.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj228 datafile 2021-07-26T06:19:29.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:29.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj229 datafile 2021-07-26T06:19:29.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:29.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj230 datafile 2021-07-26T06:19:29.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:29.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj231 datafile 2021-07-26T06:19:29.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:29.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj232 datafile 2021-07-26T06:19:29.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:29.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj233 datafile 2021-07-26T06:19:30.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:30.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj234 datafile 2021-07-26T06:19:30.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:30.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj235 datafile 2021-07-26T06:19:30.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:30.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj236 datafile 2021-07-26T06:19:30.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:30.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj237 datafile 2021-07-26T06:19:30.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:30.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj238 datafile 2021-07-26T06:19:30.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:30.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj239 datafile 2021-07-26T06:19:30.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:30.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj240 datafile 2021-07-26T06:19:31.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:31.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj241 datafile 2021-07-26T06:19:31.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:31.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj242 datafile 2021-07-26T06:19:31.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:31.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj243 datafile 2021-07-26T06:19:31.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:31.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj244 datafile 2021-07-26T06:19:31.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:31.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj245 datafile 2021-07-26T06:19:31.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:31.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj246 datafile 2021-07-26T06:19:31.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:31.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj247 datafile 2021-07-26T06:19:32.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:32.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj248 datafile 2021-07-26T06:19:32.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:32.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj249 datafile 2021-07-26T06:19:32.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:32.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj250 datafile 2021-07-26T06:19:32.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:32.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj251 datafile 2021-07-26T06:19:32.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:32.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj252 datafile 2021-07-26T06:19:32.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:32.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj253 datafile 2021-07-26T06:19:32.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:32.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj254 datafile 2021-07-26T06:19:32.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:32.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj255 datafile 2021-07-26T06:19:33.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:33.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj256 datafile 2021-07-26T06:19:33.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:33.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj257 datafile 2021-07-26T06:19:33.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:33.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj258 datafile 2021-07-26T06:19:33.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:33.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj259 datafile 2021-07-26T06:19:33.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:33.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj260 datafile 2021-07-26T06:19:33.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:33.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj261 datafile 2021-07-26T06:19:33.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:33.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj262 datafile 2021-07-26T06:19:34.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:34.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj263 datafile 2021-07-26T06:19:34.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:34.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj264 datafile 2021-07-26T06:19:34.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:34.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj265 datafile 2021-07-26T06:19:34.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:34.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj266 datafile 2021-07-26T06:19:34.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:34.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj267 datafile 2021-07-26T06:19:34.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:34.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj268 datafile 2021-07-26T06:19:34.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:34.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj269 datafile 2021-07-26T06:19:35.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:35.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj270 datafile 2021-07-26T06:19:35.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:35.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj271 datafile 2021-07-26T06:19:35.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:35.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj272 datafile 2021-07-26T06:19:35.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:35.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj273 datafile 2021-07-26T06:19:35.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:35.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj274 datafile 2021-07-26T06:19:35.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:35.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj275 datafile 2021-07-26T06:19:35.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:35.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj276 datafile 2021-07-26T06:19:36.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:36.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj277 datafile 2021-07-26T06:19:36.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:36.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj278 datafile 2021-07-26T06:19:36.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:36.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj279 datafile 2021-07-26T06:19:36.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:36.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj280 datafile 2021-07-26T06:19:36.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:36.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj281 datafile 2021-07-26T06:19:36.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:36.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj282 datafile 2021-07-26T06:19:36.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:36.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj283 datafile 2021-07-26T06:19:37.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:37.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj284 datafile 2021-07-26T06:19:37.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:37.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj285 datafile 2021-07-26T06:19:37.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:37.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj286 datafile 2021-07-26T06:19:37.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:37.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj287 datafile 2021-07-26T06:19:37.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:37.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj288 datafile 2021-07-26T06:19:37.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:37.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj289 datafile 2021-07-26T06:19:37.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:37.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj290 datafile 2021-07-26T06:19:38.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:38.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj291 datafile 2021-07-26T06:19:38.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:38.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj292 datafile 2021-07-26T06:19:38.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:38.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj293 datafile 2021-07-26T06:19:38.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:38.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj294 datafile 2021-07-26T06:19:38.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:38.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj295 datafile 2021-07-26T06:19:38.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:38.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj296 datafile 2021-07-26T06:19:38.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:38.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj297 datafile 2021-07-26T06:19:39.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:39.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj298 datafile 2021-07-26T06:19:39.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:39.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj299 datafile 2021-07-26T06:19:39.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:39.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj300 datafile 2021-07-26T06:19:39.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:39.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj301 datafile 2021-07-26T06:19:39.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:39.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj302 datafile 2021-07-26T06:19:39.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:39.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj303 datafile 2021-07-26T06:19:39.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:39.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj304 datafile 2021-07-26T06:19:40.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:40.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj305 datafile 2021-07-26T06:19:40.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:40.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj306 datafile 2021-07-26T06:19:40.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:40.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj307 datafile 2021-07-26T06:19:40.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:40.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj308 datafile 2021-07-26T06:19:40.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:40.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj309 datafile 2021-07-26T06:19:40.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:40.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj310 datafile 2021-07-26T06:19:41.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:41.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj311 datafile 2021-07-26T06:19:41.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:41.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj312 datafile 2021-07-26T06:19:41.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:41.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj313 datafile 2021-07-26T06:19:41.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:41.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj314 datafile 2021-07-26T06:19:41.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:41.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj315 datafile 2021-07-26T06:19:41.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:41.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj316 datafile 2021-07-26T06:19:41.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:41.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj317 datafile 2021-07-26T06:19:41.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:41.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj318 datafile 2021-07-26T06:19:42.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:42.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj319 datafile 2021-07-26T06:19:42.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:42.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj320 datafile 2021-07-26T06:19:42.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:42.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj321 datafile 2021-07-26T06:19:42.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:42.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj322 datafile 2021-07-26T06:19:42.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:42.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj323 datafile 2021-07-26T06:19:42.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:42.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj324 datafile 2021-07-26T06:19:42.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:42.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj325 datafile 2021-07-26T06:19:43.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:43.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj326 datafile 2021-07-26T06:19:43.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:43.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj327 datafile 2021-07-26T06:19:43.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:43.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj328 datafile 2021-07-26T06:19:43.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:43.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj329 datafile 2021-07-26T06:19:43.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:43.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj330 datafile 2021-07-26T06:19:43.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:43.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj331 datafile 2021-07-26T06:19:43.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:43.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj332 datafile 2021-07-26T06:19:44.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:44.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj333 datafile 2021-07-26T06:19:44.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:44.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj334 datafile 2021-07-26T06:19:44.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:44.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj335 datafile 2021-07-26T06:19:44.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:44.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj336 datafile 2021-07-26T06:19:44.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:44.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj337 datafile 2021-07-26T06:19:44.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:44.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj338 datafile 2021-07-26T06:19:44.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:44.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj339 datafile 2021-07-26T06:19:45.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:45.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj340 datafile 2021-07-26T06:19:45.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:45.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj341 datafile 2021-07-26T06:19:45.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:45.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj342 datafile 2021-07-26T06:19:45.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:45.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj343 datafile 2021-07-26T06:19:45.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:45.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj344 datafile 2021-07-26T06:19:45.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:45.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj345 datafile 2021-07-26T06:19:45.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:45.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj346 datafile 2021-07-26T06:19:46.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:46.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj347 datafile 2021-07-26T06:19:46.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:46.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj348 datafile 2021-07-26T06:19:46.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:46.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj349 datafile 2021-07-26T06:19:46.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:46.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj350 datafile 2021-07-26T06:19:46.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:46.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj351 datafile 2021-07-26T06:19:46.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:46.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj352 datafile 2021-07-26T06:19:46.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:46.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj353 datafile 2021-07-26T06:19:47.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:47.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj354 datafile 2021-07-26T06:19:47.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:47.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj355 datafile 2021-07-26T06:19:47.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:47.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj356 datafile 2021-07-26T06:19:47.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:47.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj357 datafile 2021-07-26T06:19:47.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:47.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj358 datafile 2021-07-26T06:19:47.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:47.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj359 datafile 2021-07-26T06:19:47.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:47.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj360 datafile 2021-07-26T06:19:48.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:48.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj361 datafile 2021-07-26T06:19:48.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:48.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj362 datafile 2021-07-26T06:19:48.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:48.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj363 datafile 2021-07-26T06:19:48.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:48.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj364 datafile 2021-07-26T06:19:48.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:48.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj365 datafile 2021-07-26T06:19:48.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:48.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj366 datafile 2021-07-26T06:19:48.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:48.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj367 datafile 2021-07-26T06:19:49.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:49.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj368 datafile 2021-07-26T06:19:49.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:49.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj369 datafile 2021-07-26T06:19:49.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:49.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj370 datafile 2021-07-26T06:19:49.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:49.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj371 datafile 2021-07-26T06:19:49.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:49.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj372 datafile 2021-07-26T06:19:49.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:49.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj373 datafile 2021-07-26T06:19:49.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:49.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj374 datafile 2021-07-26T06:19:50.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:50.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj375 datafile 2021-07-26T06:19:50.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:50.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj376 datafile 2021-07-26T06:19:50.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:50.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj377 datafile 2021-07-26T06:19:50.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:50.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj378 datafile 2021-07-26T06:19:50.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:50.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj379 datafile 2021-07-26T06:19:50.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:50.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj380 datafile 2021-07-26T06:19:50.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:50.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj381 datafile 2021-07-26T06:19:51.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:51.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj382 datafile 2021-07-26T06:19:51.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:51.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj383 datafile 2021-07-26T06:19:51.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:51.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj384 datafile 2021-07-26T06:19:51.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:51.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj385 datafile 2021-07-26T06:19:51.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:51.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj386 datafile 2021-07-26T06:19:51.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:51.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj387 datafile 2021-07-26T06:19:51.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:51.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj388 datafile 2021-07-26T06:19:52.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:52.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj389 datafile 2021-07-26T06:19:52.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:52.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj390 datafile 2021-07-26T06:19:52.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:52.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj391 datafile 2021-07-26T06:19:52.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:52.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj392 datafile 2021-07-26T06:19:52.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:52.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj393 datafile 2021-07-26T06:19:52.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:52.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj394 datafile 2021-07-26T06:19:52.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:52.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj395 datafile 2021-07-26T06:19:53.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:53.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj396 datafile 2021-07-26T06:19:53.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:53.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj397 datafile 2021-07-26T06:19:53.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:53.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj398 datafile 2021-07-26T06:19:53.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:53.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj399 datafile 2021-07-26T06:19:53.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:53.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj400 datafile 2021-07-26T06:19:53.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:53.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj401 datafile 2021-07-26T06:19:53.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:53.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj402 datafile 2021-07-26T06:19:53.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:53.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj403 datafile 2021-07-26T06:19:54.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:54.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj404 datafile 2021-07-26T06:19:54.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:54.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj405 datafile 2021-07-26T06:19:54.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:54.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj406 datafile 2021-07-26T06:19:54.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:54.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj407 datafile 2021-07-26T06:19:54.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:54.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj408 datafile 2021-07-26T06:19:54.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:54.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj409 datafile 2021-07-26T06:19:54.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:54.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj410 datafile 2021-07-26T06:19:55.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:55.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj411 datafile 2021-07-26T06:19:55.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:55.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj412 datafile 2021-07-26T06:19:55.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:55.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj413 datafile 2021-07-26T06:19:55.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:55.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj414 datafile 2021-07-26T06:19:55.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:55.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj415 datafile 2021-07-26T06:19:56.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:56.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj416 datafile 2021-07-26T06:19:56.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:56.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj417 datafile 2021-07-26T06:19:56.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:56.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj418 datafile 2021-07-26T06:19:56.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:56.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj419 datafile 2021-07-26T06:19:56.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:56.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj420 datafile 2021-07-26T06:19:56.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:56.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj421 datafile 2021-07-26T06:19:57.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:57.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj422 datafile 2021-07-26T06:19:57.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:57.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj423 datafile 2021-07-26T06:19:57.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:57.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj424 datafile 2021-07-26T06:19:57.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:57.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj425 datafile 2021-07-26T06:19:57.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:57.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj426 datafile 2021-07-26T06:19:57.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:57.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj427 datafile 2021-07-26T06:19:57.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:57.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj428 datafile 2021-07-26T06:19:57.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:57.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj429 datafile 2021-07-26T06:19:58.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:58.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj430 datafile 2021-07-26T06:19:58.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:58.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj431 datafile 2021-07-26T06:19:58.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:58.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj432 datafile 2021-07-26T06:19:58.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:58.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj433 datafile 2021-07-26T06:19:58.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:58.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj434 datafile 2021-07-26T06:19:58.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:58.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj435 datafile 2021-07-26T06:19:58.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:58.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj436 datafile 2021-07-26T06:19:59.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:59.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj437 datafile 2021-07-26T06:19:59.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:59.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj438 datafile 2021-07-26T06:19:59.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:59.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj439 datafile 2021-07-26T06:19:59.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:59.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj440 datafile 2021-07-26T06:19:59.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:59.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj441 datafile 2021-07-26T06:19:59.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:59.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj442 datafile 2021-07-26T06:19:59.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:19:59.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj443 datafile 2021-07-26T06:20:00.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:00.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj444 datafile 2021-07-26T06:20:00.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:00.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj445 datafile 2021-07-26T06:20:00.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:00.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj446 datafile 2021-07-26T06:20:00.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:00.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj447 datafile 2021-07-26T06:20:00.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:00.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj448 datafile 2021-07-26T06:20:00.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:00.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj449 datafile 2021-07-26T06:20:01.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:01.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj450 datafile 2021-07-26T06:20:01.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:01.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj451 datafile 2021-07-26T06:20:01.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:01.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj452 datafile 2021-07-26T06:20:01.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:01.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj453 datafile 2021-07-26T06:20:01.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:01.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj454 datafile 2021-07-26T06:20:01.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:01.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj455 datafile 2021-07-26T06:20:01.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:01.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj456 datafile 2021-07-26T06:20:02.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:02.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj457 datafile 2021-07-26T06:20:02.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:02.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj458 datafile 2021-07-26T06:20:02.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:02.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj459 datafile 2021-07-26T06:20:02.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:02.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj460 datafile 2021-07-26T06:20:02.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:02.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj461 datafile 2021-07-26T06:20:02.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:02.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj462 datafile 2021-07-26T06:20:02.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:02.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj463 datafile 2021-07-26T06:20:03.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:03.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj464 datafile 2021-07-26T06:20:03.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:03.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj465 datafile 2021-07-26T06:20:03.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:03.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj466 datafile 2021-07-26T06:20:03.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:03.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj467 datafile 2021-07-26T06:20:03.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:03.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj468 datafile 2021-07-26T06:20:03.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:03.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj469 datafile 2021-07-26T06:20:03.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:03.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj470 datafile 2021-07-26T06:20:04.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:04.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj471 datafile 2021-07-26T06:20:04.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:04.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj472 datafile 2021-07-26T06:20:04.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:04.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj473 datafile 2021-07-26T06:20:04.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:04.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj474 datafile 2021-07-26T06:20:04.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:04.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj475 datafile 2021-07-26T06:20:04.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:04.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj476 datafile 2021-07-26T06:20:04.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:04.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj477 datafile 2021-07-26T06:20:05.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:05.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj478 datafile 2021-07-26T06:20:05.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:05.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj479 datafile 2021-07-26T06:20:05.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:05.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj480 datafile 2021-07-26T06:20:05.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:05.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj481 datafile 2021-07-26T06:20:05.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:05.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj482 datafile 2021-07-26T06:20:05.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:05.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj483 datafile 2021-07-26T06:20:05.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:05.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj484 datafile 2021-07-26T06:20:06.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:06.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj485 datafile 2021-07-26T06:20:06.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:06.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj486 datafile 2021-07-26T06:20:06.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:06.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj487 datafile 2021-07-26T06:20:06.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:06.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj488 datafile 2021-07-26T06:20:06.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:06.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj489 datafile 2021-07-26T06:20:07.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:07.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj490 datafile 2021-07-26T06:20:07.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:07.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj491 datafile 2021-07-26T06:20:07.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:07.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj492 datafile 2021-07-26T06:20:07.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:07.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj493 datafile 2021-07-26T06:20:07.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:07.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj494 datafile 2021-07-26T06:20:07.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:07.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj495 datafile 2021-07-26T06:20:07.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:07.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj496 datafile 2021-07-26T06:20:07.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:07.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj497 datafile 2021-07-26T06:20:08.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:08.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj498 datafile 2021-07-26T06:20:08.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:08.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj499 datafile 2021-07-26T06:20:08.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:08.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj500 datafile 2021-07-26T06:20:08.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:08.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj501 datafile 2021-07-26T06:20:08.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:08.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj502 datafile 2021-07-26T06:20:08.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:08.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj503 datafile 2021-07-26T06:20:09.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:09.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj504 datafile 2021-07-26T06:20:09.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:09.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj505 datafile 2021-07-26T06:20:09.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:09.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj506 datafile 2021-07-26T06:20:09.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:09.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj507 datafile 2021-07-26T06:20:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj508 datafile 2021-07-26T06:20:09.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:09.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj509 datafile 2021-07-26T06:20:09.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:09.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj510 datafile 2021-07-26T06:20:10.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:10.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj511 datafile 2021-07-26T06:20:10.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:10.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj512 datafile 2021-07-26T06:20:10.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:10.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj513 datafile 2021-07-26T06:20:10.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:10.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj514 datafile 2021-07-26T06:20:10.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:10.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj515 datafile 2021-07-26T06:20:10.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:10.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj516 datafile 2021-07-26T06:20:10.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:10.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj517 datafile 2021-07-26T06:20:10.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:10.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj518 datafile 2021-07-26T06:20:11.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:11.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj519 datafile 2021-07-26T06:20:11.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:11.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj520 datafile 2021-07-26T06:20:11.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:11.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj521 datafile 2021-07-26T06:20:11.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:11.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj522 datafile 2021-07-26T06:20:11.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:11.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj523 datafile 2021-07-26T06:20:11.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:11.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj524 datafile 2021-07-26T06:20:12.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:12.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj525 datafile 2021-07-26T06:20:12.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:12.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj526 datafile 2021-07-26T06:20:12.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:12.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj527 datafile 2021-07-26T06:20:12.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:12.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj528 datafile 2021-07-26T06:20:12.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:12.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj529 datafile 2021-07-26T06:20:12.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:12.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj530 datafile 2021-07-26T06:20:12.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:12.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj531 datafile 2021-07-26T06:20:12.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:12.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj532 datafile 2021-07-26T06:20:13.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:13.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj533 datafile 2021-07-26T06:20:13.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:13.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj534 datafile 2021-07-26T06:20:13.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:13.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj535 datafile 2021-07-26T06:20:13.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:13.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj536 datafile 2021-07-26T06:20:13.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:13.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj537 datafile 2021-07-26T06:20:13.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:13.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj538 datafile 2021-07-26T06:20:14.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:14.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj539 datafile 2021-07-26T06:20:14.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:14.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj540 datafile 2021-07-26T06:20:14.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:14.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj541 datafile 2021-07-26T06:20:14.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:14.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj542 datafile 2021-07-26T06:20:14.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:14.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj543 datafile 2021-07-26T06:20:14.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:14.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj544 datafile 2021-07-26T06:20:14.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:14.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj545 datafile 2021-07-26T06:20:14.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:14.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj546 datafile 2021-07-26T06:20:15.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:15.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj547 datafile 2021-07-26T06:20:15.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:15.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj548 datafile 2021-07-26T06:20:15.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:15.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj549 datafile 2021-07-26T06:20:15.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:15.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj550 datafile 2021-07-26T06:20:15.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:15.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj551 datafile 2021-07-26T06:20:15.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:15.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj552 datafile 2021-07-26T06:20:16.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:16.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj553 datafile 2021-07-26T06:20:16.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:16.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj554 datafile 2021-07-26T06:20:16.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:16.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj555 datafile 2021-07-26T06:20:16.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:16.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj556 datafile 2021-07-26T06:20:16.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:16.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj557 datafile 2021-07-26T06:20:16.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:16.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj558 datafile 2021-07-26T06:20:16.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:16.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj559 datafile 2021-07-26T06:20:16.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:16.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj560 datafile 2021-07-26T06:20:17.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:17.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj561 datafile 2021-07-26T06:20:17.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:17.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj562 datafile 2021-07-26T06:20:17.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:17.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj563 datafile 2021-07-26T06:20:17.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:17.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj564 datafile 2021-07-26T06:20:17.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:17.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj565 datafile 2021-07-26T06:20:17.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:17.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj566 datafile 2021-07-26T06:20:17.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:17.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj567 datafile 2021-07-26T06:20:18.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:18.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj568 datafile 2021-07-26T06:20:18.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:18.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj569 datafile 2021-07-26T06:20:18.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:18.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj570 datafile 2021-07-26T06:20:18.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:18.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj571 datafile 2021-07-26T06:20:18.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:18.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj572 datafile 2021-07-26T06:20:18.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:18.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj573 datafile 2021-07-26T06:20:18.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:18.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj574 datafile 2021-07-26T06:20:19.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:19.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj575 datafile 2021-07-26T06:20:19.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:19.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj576 datafile 2021-07-26T06:20:19.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:19.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj577 datafile 2021-07-26T06:20:19.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:19.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj578 datafile 2021-07-26T06:20:19.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:19.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj579 datafile 2021-07-26T06:20:19.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:19.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj580 datafile 2021-07-26T06:20:19.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:19.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj581 datafile 2021-07-26T06:20:20.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:20.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj582 datafile 2021-07-26T06:20:20.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:20.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj583 datafile 2021-07-26T06:20:20.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:20.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj584 datafile 2021-07-26T06:20:20.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:20.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj585 datafile 2021-07-26T06:20:20.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:20.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj586 datafile 2021-07-26T06:20:20.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:20.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj587 datafile 2021-07-26T06:20:20.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:20.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj588 datafile 2021-07-26T06:20:21.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:21.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj589 datafile 2021-07-26T06:20:21.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:21.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj590 datafile 2021-07-26T06:20:21.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:21.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj591 datafile 2021-07-26T06:20:21.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:21.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj592 datafile 2021-07-26T06:20:21.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:21.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj593 datafile 2021-07-26T06:20:21.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:21.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj594 datafile 2021-07-26T06:20:21.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:21.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj595 datafile 2021-07-26T06:20:22.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:22.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj596 datafile 2021-07-26T06:20:22.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:22.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj597 datafile 2021-07-26T06:20:22.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:22.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj598 datafile 2021-07-26T06:20:22.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:22.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj599 datafile 2021-07-26T06:20:22.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:22.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj600 datafile 2021-07-26T06:20:22.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:22.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj601 datafile 2021-07-26T06:20:22.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:22.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj602 datafile 2021-07-26T06:20:23.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:23.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj603 datafile 2021-07-26T06:20:23.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:23.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj604 datafile 2021-07-26T06:20:23.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:23.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj605 datafile 2021-07-26T06:20:23.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:23.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj606 datafile 2021-07-26T06:20:23.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:23.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj607 datafile 2021-07-26T06:20:23.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:23.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj608 datafile 2021-07-26T06:20:24.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:24.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj609 datafile 2021-07-26T06:20:24.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:24.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj610 datafile 2021-07-26T06:20:24.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:24.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj611 datafile 2021-07-26T06:20:24.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:24.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj612 datafile 2021-07-26T06:20:24.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:24.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj613 datafile 2021-07-26T06:20:24.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:24.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj614 datafile 2021-07-26T06:20:24.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:24.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj615 datafile 2021-07-26T06:20:25.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:25.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj616 datafile 2021-07-26T06:20:25.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:25.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj617 datafile 2021-07-26T06:20:25.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:25.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj618 datafile 2021-07-26T06:20:25.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:25.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj619 datafile 2021-07-26T06:20:25.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:25.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj620 datafile 2021-07-26T06:20:25.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:25.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj621 datafile 2021-07-26T06:20:25.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:25.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj622 datafile 2021-07-26T06:20:26.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:26.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj623 datafile 2021-07-26T06:20:26.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:26.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj624 datafile 2021-07-26T06:20:26.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:26.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj625 datafile 2021-07-26T06:20:26.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:26.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj626 datafile 2021-07-26T06:20:26.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:26.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj627 datafile 2021-07-26T06:20:26.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:26.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj628 datafile 2021-07-26T06:20:26.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:26.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj629 datafile 2021-07-26T06:20:27.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:27.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj630 datafile 2021-07-26T06:20:27.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:27.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj631 datafile 2021-07-26T06:20:27.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:27.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj632 datafile 2021-07-26T06:20:27.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:27.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj633 datafile 2021-07-26T06:20:27.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:27.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj634 datafile 2021-07-26T06:20:28.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:28.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj635 datafile 2021-07-26T06:20:28.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:28.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj636 datafile 2021-07-26T06:20:28.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:28.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj637 datafile 2021-07-26T06:20:28.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:28.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj638 datafile 2021-07-26T06:20:28.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:28.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj639 datafile 2021-07-26T06:20:28.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:28.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj640 datafile 2021-07-26T06:20:28.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:28.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj641 datafile 2021-07-26T06:20:29.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:29.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj642 datafile 2021-07-26T06:20:29.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:29.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj643 datafile 2021-07-26T06:20:29.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:29.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj644 datafile 2021-07-26T06:20:29.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:29.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj645 datafile 2021-07-26T06:20:29.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:29.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj646 datafile 2021-07-26T06:20:29.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:29.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj647 datafile 2021-07-26T06:20:29.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:29.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj648 datafile 2021-07-26T06:20:30.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:30.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj649 datafile 2021-07-26T06:20:30.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:30.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj650 datafile 2021-07-26T06:20:30.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:30.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj651 datafile 2021-07-26T06:20:30.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:30.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj652 datafile 2021-07-26T06:20:30.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:30.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj653 datafile 2021-07-26T06:20:30.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:30.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj654 datafile 2021-07-26T06:20:30.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:30.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj655 datafile 2021-07-26T06:20:31.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:31.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj656 datafile 2021-07-26T06:20:31.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:31.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj657 datafile 2021-07-26T06:20:31.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:31.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj658 datafile 2021-07-26T06:20:31.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:31.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj659 datafile 2021-07-26T06:20:31.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:31.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj660 datafile 2021-07-26T06:20:31.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:31.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj661 datafile 2021-07-26T06:20:31.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:31.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj662 datafile 2021-07-26T06:20:32.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:32.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj663 datafile 2021-07-26T06:20:32.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:32.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj664 datafile 2021-07-26T06:20:32.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:32.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj665 datafile 2021-07-26T06:20:32.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:32.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj666 datafile 2021-07-26T06:20:32.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:32.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj667 datafile 2021-07-26T06:20:32.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:32.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj668 datafile 2021-07-26T06:20:33.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:33.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj669 datafile 2021-07-26T06:20:33.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:33.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj670 datafile 2021-07-26T06:20:33.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:33.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj671 datafile 2021-07-26T06:20:33.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:33.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj672 datafile 2021-07-26T06:20:33.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:33.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj673 datafile 2021-07-26T06:20:33.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:33.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj674 datafile 2021-07-26T06:20:33.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:33.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj675 datafile 2021-07-26T06:20:34.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:34.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj676 datafile 2021-07-26T06:20:34.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:34.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj677 datafile 2021-07-26T06:20:34.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:34.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj678 datafile 2021-07-26T06:20:34.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:34.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj679 datafile 2021-07-26T06:20:34.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:34.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj680 datafile 2021-07-26T06:20:34.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:34.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj681 datafile 2021-07-26T06:20:34.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:34.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj682 datafile 2021-07-26T06:20:35.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:35.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj683 datafile 2021-07-26T06:20:35.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:35.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj684 datafile 2021-07-26T06:20:35.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:35.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj685 datafile 2021-07-26T06:20:35.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:35.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj686 datafile 2021-07-26T06:20:35.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:35.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj687 datafile 2021-07-26T06:20:35.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:35.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj688 datafile 2021-07-26T06:20:35.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:35.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj689 datafile 2021-07-26T06:20:36.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:36.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj690 datafile 2021-07-26T06:20:36.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:36.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj691 datafile 2021-07-26T06:20:36.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:36.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj692 datafile 2021-07-26T06:20:36.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:36.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj693 datafile 2021-07-26T06:20:36.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:36.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj694 datafile 2021-07-26T06:20:36.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:36.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj695 datafile 2021-07-26T06:20:36.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:36.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj696 datafile 2021-07-26T06:20:36.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:36.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj697 datafile 2021-07-26T06:20:37.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:37.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj698 datafile 2021-07-26T06:20:37.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:37.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj699 datafile 2021-07-26T06:20:37.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:37.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj700 datafile 2021-07-26T06:20:37.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:37.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj701 datafile 2021-07-26T06:20:37.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:37.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj702 datafile 2021-07-26T06:20:37.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:37.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj703 datafile 2021-07-26T06:20:38.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:38.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj704 datafile 2021-07-26T06:20:38.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:38.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj705 datafile 2021-07-26T06:20:38.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:38.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj706 datafile 2021-07-26T06:20:38.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:38.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj707 datafile 2021-07-26T06:20:38.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:38.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj708 datafile 2021-07-26T06:20:38.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:38.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj709 datafile 2021-07-26T06:20:38.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:38.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj710 datafile 2021-07-26T06:20:39.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:39.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj711 datafile 2021-07-26T06:20:39.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:39.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj712 datafile 2021-07-26T06:20:39.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:39.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj713 datafile 2021-07-26T06:20:39.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:39.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj714 datafile 2021-07-26T06:20:39.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:39.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj715 datafile 2021-07-26T06:20:39.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:39.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj716 datafile 2021-07-26T06:20:40.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:40.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj717 datafile 2021-07-26T06:20:40.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:40.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj718 datafile 2021-07-26T06:20:40.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:40.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj719 datafile 2021-07-26T06:20:40.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:40.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj720 datafile 2021-07-26T06:20:40.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:40.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj721 datafile 2021-07-26T06:20:40.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:40.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj722 datafile 2021-07-26T06:20:40.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:40.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj723 datafile 2021-07-26T06:20:41.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:41.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj724 datafile 2021-07-26T06:20:41.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:41.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj725 datafile 2021-07-26T06:20:41.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:41.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj726 datafile 2021-07-26T06:20:41.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:41.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj727 datafile 2021-07-26T06:20:41.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:41.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj728 datafile 2021-07-26T06:20:41.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:41.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj729 datafile 2021-07-26T06:20:41.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:41.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj730 datafile 2021-07-26T06:20:42.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:42.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj731 datafile 2021-07-26T06:20:42.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:42.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj732 datafile 2021-07-26T06:20:42.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:42.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj733 datafile 2021-07-26T06:20:42.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:42.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj734 datafile 2021-07-26T06:20:42.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:42.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj735 datafile 2021-07-26T06:20:42.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:42.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj736 datafile 2021-07-26T06:20:42.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:42.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj737 datafile 2021-07-26T06:20:43.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:43.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj738 datafile 2021-07-26T06:20:43.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:43.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj739 datafile 2021-07-26T06:20:43.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:43.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj740 datafile 2021-07-26T06:20:43.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:43.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj741 datafile 2021-07-26T06:20:43.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:43.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj742 datafile 2021-07-26T06:20:43.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:43.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj743 datafile 2021-07-26T06:20:43.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:43.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj744 datafile 2021-07-26T06:20:44.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:44.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj745 datafile 2021-07-26T06:20:44.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:44.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj746 datafile 2021-07-26T06:20:44.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:44.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj747 datafile 2021-07-26T06:20:44.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:44.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj748 datafile 2021-07-26T06:20:44.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:44.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj749 datafile 2021-07-26T06:20:44.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:44.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj750 datafile 2021-07-26T06:20:44.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:44.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj751 datafile 2021-07-26T06:20:45.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:45.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj752 datafile 2021-07-26T06:20:45.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:45.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj753 datafile 2021-07-26T06:20:45.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:45.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj754 datafile 2021-07-26T06:20:45.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:45.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj755 datafile 2021-07-26T06:20:45.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:45.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj756 datafile 2021-07-26T06:20:45.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:45.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj757 datafile 2021-07-26T06:20:45.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:45.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj758 datafile 2021-07-26T06:20:45.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:45.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj759 datafile 2021-07-26T06:20:46.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:46.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj760 datafile 2021-07-26T06:20:46.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:46.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj761 datafile 2021-07-26T06:20:46.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:46.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj762 datafile 2021-07-26T06:20:46.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:46.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj763 datafile 2021-07-26T06:20:46.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:46.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj764 datafile 2021-07-26T06:20:46.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:46.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj765 datafile 2021-07-26T06:20:46.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:46.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj766 datafile 2021-07-26T06:20:47.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:47.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj767 datafile 2021-07-26T06:20:47.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:47.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj768 datafile 2021-07-26T06:20:47.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:47.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj769 datafile 2021-07-26T06:20:47.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:47.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj770 datafile 2021-07-26T06:20:47.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:47.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj771 datafile 2021-07-26T06:20:47.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:47.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj772 datafile 2021-07-26T06:20:47.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:47.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj773 datafile 2021-07-26T06:20:48.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:48.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj774 datafile 2021-07-26T06:20:48.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:48.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj775 datafile 2021-07-26T06:20:48.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:48.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj776 datafile 2021-07-26T06:20:48.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:48.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj777 datafile 2021-07-26T06:20:48.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:48.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj778 datafile 2021-07-26T06:20:48.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:48.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj779 datafile 2021-07-26T06:20:48.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:48.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj780 datafile 2021-07-26T06:20:49.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:49.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj781 datafile 2021-07-26T06:20:49.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:49.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj782 datafile 2021-07-26T06:20:49.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:49.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj783 datafile 2021-07-26T06:20:49.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:49.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj784 datafile 2021-07-26T06:20:49.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:49.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj785 datafile 2021-07-26T06:20:49.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:49.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj786 datafile 2021-07-26T06:20:50.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:50.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj787 datafile 2021-07-26T06:20:50.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:50.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj788 datafile 2021-07-26T06:20:50.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:50.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj789 datafile 2021-07-26T06:20:50.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:50.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj790 datafile 2021-07-26T06:20:50.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:50.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj791 datafile 2021-07-26T06:20:50.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:50.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj792 datafile 2021-07-26T06:20:50.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:50.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj793 datafile 2021-07-26T06:20:50.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:50.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj794 datafile 2021-07-26T06:20:51.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:51.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj795 datafile 2021-07-26T06:20:51.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:51.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj796 datafile 2021-07-26T06:20:51.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:51.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj797 datafile 2021-07-26T06:20:51.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:51.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj798 datafile 2021-07-26T06:20:51.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:51.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj799 datafile 2021-07-26T06:20:51.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:51.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj800 datafile 2021-07-26T06:20:52.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:52.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj801 datafile 2021-07-26T06:20:52.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:52.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj802 datafile 2021-07-26T06:20:52.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:52.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj803 datafile 2021-07-26T06:20:52.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:52.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj804 datafile 2021-07-26T06:20:52.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:52.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj805 datafile 2021-07-26T06:20:52.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:52.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj806 datafile 2021-07-26T06:20:52.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:52.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj807 datafile 2021-07-26T06:20:52.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:52.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj808 datafile 2021-07-26T06:20:53.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:53.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj809 datafile 2021-07-26T06:20:53.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:53.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj810 datafile 2021-07-26T06:20:53.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:53.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj811 datafile 2021-07-26T06:20:53.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:53.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj812 datafile 2021-07-26T06:20:53.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:53.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj813 datafile 2021-07-26T06:20:53.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:53.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj814 datafile 2021-07-26T06:20:53.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:53.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj815 datafile 2021-07-26T06:20:54.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:54.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj816 datafile 2021-07-26T06:20:54.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:54.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj817 datafile 2021-07-26T06:20:54.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:54.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj818 datafile 2021-07-26T06:20:54.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:54.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj819 datafile 2021-07-26T06:20:54.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:54.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj820 datafile 2021-07-26T06:20:54.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:54.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj821 datafile 2021-07-26T06:20:54.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:54.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj822 datafile 2021-07-26T06:20:55.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:55.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj823 datafile 2021-07-26T06:20:55.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:55.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj824 datafile 2021-07-26T06:20:55.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:55.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj825 datafile 2021-07-26T06:20:55.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:55.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj826 datafile 2021-07-26T06:20:55.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:55.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj827 datafile 2021-07-26T06:20:55.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:55.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj828 datafile 2021-07-26T06:20:56.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:56.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj829 datafile 2021-07-26T06:20:56.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:56.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj830 datafile 2021-07-26T06:20:56.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:56.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj831 datafile 2021-07-26T06:20:56.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:56.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj832 datafile 2021-07-26T06:20:56.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:56.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj833 datafile 2021-07-26T06:20:56.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:56.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj834 datafile 2021-07-26T06:20:56.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:56.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj835 datafile 2021-07-26T06:20:57.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:57.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj836 datafile 2021-07-26T06:20:57.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:57.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj837 datafile 2021-07-26T06:20:57.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:57.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj838 datafile 2021-07-26T06:20:57.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:57.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj839 datafile 2021-07-26T06:20:57.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:57.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj840 datafile 2021-07-26T06:20:57.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:57.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj841 datafile 2021-07-26T06:20:57.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:57.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj842 datafile 2021-07-26T06:20:58.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:58.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj843 datafile 2021-07-26T06:20:58.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:58.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj844 datafile 2021-07-26T06:20:58.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:58.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj845 datafile 2021-07-26T06:20:58.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:58.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj846 datafile 2021-07-26T06:20:58.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:58.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj847 datafile 2021-07-26T06:20:58.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:58.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj848 datafile 2021-07-26T06:20:59.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:59.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj849 datafile 2021-07-26T06:20:59.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:59.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj850 datafile 2021-07-26T06:20:59.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:59.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj851 datafile 2021-07-26T06:20:59.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:59.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj852 datafile 2021-07-26T06:20:59.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:59.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj853 datafile 2021-07-26T06:20:59.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:20:59.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj854 datafile 2021-07-26T06:21:00.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:00.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj855 datafile 2021-07-26T06:21:00.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:00.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj856 datafile 2021-07-26T06:21:00.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:00.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj857 datafile 2021-07-26T06:21:00.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:00.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj858 datafile 2021-07-26T06:21:00.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:00.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj859 datafile 2021-07-26T06:21:00.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:00.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj860 datafile 2021-07-26T06:21:00.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:00.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj861 datafile 2021-07-26T06:21:01.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:01.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj862 datafile 2021-07-26T06:21:01.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:01.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj863 datafile 2021-07-26T06:21:01.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:01.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj864 datafile 2021-07-26T06:21:01.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:01.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj865 datafile 2021-07-26T06:21:01.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:01.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj866 datafile 2021-07-26T06:21:01.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:01.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj867 datafile 2021-07-26T06:21:01.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:01.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj868 datafile 2021-07-26T06:21:01.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:01.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj869 datafile 2021-07-26T06:21:02.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:02.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj870 datafile 2021-07-26T06:21:02.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:02.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj871 datafile 2021-07-26T06:21:02.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:02.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj872 datafile 2021-07-26T06:21:02.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:02.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj873 datafile 2021-07-26T06:21:02.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:02.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj874 datafile 2021-07-26T06:21:02.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:02.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj875 datafile 2021-07-26T06:21:02.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:02.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj876 datafile 2021-07-26T06:21:03.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:03.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj877 datafile 2021-07-26T06:21:03.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:03.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj878 datafile 2021-07-26T06:21:03.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:03.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj879 datafile 2021-07-26T06:21:03.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:03.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj880 datafile 2021-07-26T06:21:03.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:03.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj881 datafile 2021-07-26T06:21:03.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:03.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj882 datafile 2021-07-26T06:21:03.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:03.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj883 datafile 2021-07-26T06:21:04.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:04.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj884 datafile 2021-07-26T06:21:04.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:04.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj885 datafile 2021-07-26T06:21:04.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:04.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj886 datafile 2021-07-26T06:21:04.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:04.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj887 datafile 2021-07-26T06:21:04.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:04.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj888 datafile 2021-07-26T06:21:04.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:04.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj889 datafile 2021-07-26T06:21:04.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:04.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj890 datafile 2021-07-26T06:21:04.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:04.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj891 datafile 2021-07-26T06:21:05.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:05.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj892 datafile 2021-07-26T06:21:05.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:05.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj893 datafile 2021-07-26T06:21:05.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:05.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj894 datafile 2021-07-26T06:21:05.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:05.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj895 datafile 2021-07-26T06:21:05.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:05.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj896 datafile 2021-07-26T06:21:05.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:05.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj897 datafile 2021-07-26T06:21:05.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:05.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj898 datafile 2021-07-26T06:21:06.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:06.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj899 datafile 2021-07-26T06:21:06.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:06.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj900 datafile 2021-07-26T06:21:06.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:06.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj901 datafile 2021-07-26T06:21:06.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:06.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj902 datafile 2021-07-26T06:21:06.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:06.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj903 datafile 2021-07-26T06:21:06.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:06.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj904 datafile 2021-07-26T06:21:06.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:06.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj905 datafile 2021-07-26T06:21:07.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:07.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj906 datafile 2021-07-26T06:21:07.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:07.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj907 datafile 2021-07-26T06:21:07.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:07.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj908 datafile 2021-07-26T06:21:07.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:07.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj909 datafile 2021-07-26T06:21:07.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:07.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj910 datafile 2021-07-26T06:21:07.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:07.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj911 datafile 2021-07-26T06:21:07.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:07.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj912 datafile 2021-07-26T06:21:08.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:08.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj913 datafile 2021-07-26T06:21:08.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:08.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj914 datafile 2021-07-26T06:21:08.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:08.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj915 datafile 2021-07-26T06:21:08.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:08.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj916 datafile 2021-07-26T06:21:08.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:08.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj917 datafile 2021-07-26T06:21:08.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:08.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj918 datafile 2021-07-26T06:21:08.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:08.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj919 datafile 2021-07-26T06:21:09.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:09.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj920 datafile 2021-07-26T06:21:09.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:09.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj921 datafile 2021-07-26T06:21:09.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:09.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj922 datafile 2021-07-26T06:21:09.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:09.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj923 datafile 2021-07-26T06:21:09.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:09.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj924 datafile 2021-07-26T06:21:09.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:09.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj925 datafile 2021-07-26T06:21:09.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:09.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj926 datafile 2021-07-26T06:21:10.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:10.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj927 datafile 2021-07-26T06:21:10.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:10.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj928 datafile 2021-07-26T06:21:10.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:10.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj929 datafile 2021-07-26T06:21:10.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:10.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj930 datafile 2021-07-26T06:21:10.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:10.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj931 datafile 2021-07-26T06:21:10.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:10.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj932 datafile 2021-07-26T06:21:10.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:10.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj933 datafile 2021-07-26T06:21:11.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:11.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj934 datafile 2021-07-26T06:21:11.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:11.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj935 datafile 2021-07-26T06:21:11.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:11.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj936 datafile 2021-07-26T06:21:11.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:11.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj937 datafile 2021-07-26T06:21:11.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:11.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj938 datafile 2021-07-26T06:21:11.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:11.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj939 datafile 2021-07-26T06:21:11.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:11.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj940 datafile 2021-07-26T06:21:12.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:12.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj941 datafile 2021-07-26T06:21:12.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:12.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj942 datafile 2021-07-26T06:21:12.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:12.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj943 datafile 2021-07-26T06:21:12.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:12.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj944 datafile 2021-07-26T06:21:12.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:12.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj945 datafile 2021-07-26T06:21:12.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:12.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj946 datafile 2021-07-26T06:21:12.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:12.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj947 datafile 2021-07-26T06:21:13.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:13.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj948 datafile 2021-07-26T06:21:13.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:13.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj949 datafile 2021-07-26T06:21:13.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:13.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj950 datafile 2021-07-26T06:21:13.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:13.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj951 datafile 2021-07-26T06:21:13.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:13.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj952 datafile 2021-07-26T06:21:13.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:13.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj953 datafile 2021-07-26T06:21:13.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:13.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj954 datafile 2021-07-26T06:21:13.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:13.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj955 datafile 2021-07-26T06:21:14.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:14.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj956 datafile 2021-07-26T06:21:14.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:14.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj957 datafile 2021-07-26T06:21:14.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:14.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj958 datafile 2021-07-26T06:21:14.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:14.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj959 datafile 2021-07-26T06:21:14.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:14.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj960 datafile 2021-07-26T06:21:14.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:14.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj961 datafile 2021-07-26T06:21:14.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:14.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj962 datafile 2021-07-26T06:21:15.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:15.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj963 datafile 2021-07-26T06:21:15.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:15.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj964 datafile 2021-07-26T06:21:15.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:15.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj965 datafile 2021-07-26T06:21:15.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:15.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj966 datafile 2021-07-26T06:21:15.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:15.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj967 datafile 2021-07-26T06:21:15.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:15.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj968 datafile 2021-07-26T06:21:16.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:16.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj969 datafile 2021-07-26T06:21:16.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:16.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj970 datafile 2021-07-26T06:21:16.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:16.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj971 datafile 2021-07-26T06:21:16.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:16.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj972 datafile 2021-07-26T06:21:16.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:16.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj973 datafile 2021-07-26T06:21:16.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:16.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj974 datafile 2021-07-26T06:21:16.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:16.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj975 datafile 2021-07-26T06:21:17.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:17.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj976 datafile 2021-07-26T06:21:17.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:17.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj977 datafile 2021-07-26T06:21:17.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:17.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj978 datafile 2021-07-26T06:21:17.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:17.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj979 datafile 2021-07-26T06:21:17.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:17.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj980 datafile 2021-07-26T06:21:17.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:17.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj981 datafile 2021-07-26T06:21:17.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:17.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj982 datafile 2021-07-26T06:21:17.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:17.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj983 datafile 2021-07-26T06:21:18.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:18.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj984 datafile 2021-07-26T06:21:18.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:18.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj985 datafile 2021-07-26T06:21:18.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:18.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj986 datafile 2021-07-26T06:21:18.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:18.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj987 datafile 2021-07-26T06:21:18.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:18.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj988 datafile 2021-07-26T06:21:18.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:18.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj989 datafile 2021-07-26T06:21:18.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:18.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj990 datafile 2021-07-26T06:21:19.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:19.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj991 datafile 2021-07-26T06:21:19.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:19.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj992 datafile 2021-07-26T06:21:19.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:19.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj993 datafile 2021-07-26T06:21:19.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:19.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj994 datafile 2021-07-26T06:21:19.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:19.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj995 datafile 2021-07-26T06:21:19.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:19.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj996 datafile 2021-07-26T06:21:19.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:19.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj997 datafile 2021-07-26T06:21:20.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:20.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj998 datafile 2021-07-26T06:21:20.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:20.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj999 datafile 2021-07-26T06:21:20.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:20.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj1000 datafile 2021-07-26T06:21:20.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:72: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2021-07-26T06:21:20.525 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: seq 1 1000 2021-07-26T06:21:20.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:20.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj1 datafile 2021-07-26T06:21:20.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:20.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj2 datafile 2021-07-26T06:21:20.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:20.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj3 datafile 2021-07-26T06:21:21.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:21.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj4 datafile 2021-07-26T06:21:21.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:21.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj5 datafile 2021-07-26T06:21:21.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:21.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj6 datafile 2021-07-26T06:21:21.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:21.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj7 datafile 2021-07-26T06:21:21.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:21.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj8 datafile 2021-07-26T06:21:21.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:21.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj9 datafile 2021-07-26T06:21:21.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:21.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj10 datafile 2021-07-26T06:21:22.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:22.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj11 datafile 2021-07-26T06:21:22.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:22.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj12 datafile 2021-07-26T06:21:22.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:22.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj13 datafile 2021-07-26T06:21:22.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:22.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj14 datafile 2021-07-26T06:21:22.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:22.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj15 datafile 2021-07-26T06:21:22.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:22.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj16 datafile 2021-07-26T06:21:22.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:22.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj17 datafile 2021-07-26T06:21:23.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:23.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj18 datafile 2021-07-26T06:21:23.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:23.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj19 datafile 2021-07-26T06:21:23.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:23.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj20 datafile 2021-07-26T06:21:23.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:23.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj21 datafile 2021-07-26T06:21:23.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:23.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj22 datafile 2021-07-26T06:21:23.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:23.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj23 datafile 2021-07-26T06:21:23.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:23.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj24 datafile 2021-07-26T06:21:24.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:24.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj25 datafile 2021-07-26T06:21:24.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:24.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj26 datafile 2021-07-26T06:21:24.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:24.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj27 datafile 2021-07-26T06:21:24.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:24.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj28 datafile 2021-07-26T06:21:24.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:24.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj29 datafile 2021-07-26T06:21:24.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:24.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj30 datafile 2021-07-26T06:21:25.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:25.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj31 datafile 2021-07-26T06:21:25.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:25.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj32 datafile 2021-07-26T06:21:25.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:25.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj33 datafile 2021-07-26T06:21:25.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:25.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj34 datafile 2021-07-26T06:21:25.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:25.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj35 datafile 2021-07-26T06:21:25.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:25.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj36 datafile 2021-07-26T06:21:25.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:25.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj37 datafile 2021-07-26T06:21:26.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:26.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj38 datafile 2021-07-26T06:21:26.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:26.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj39 datafile 2021-07-26T06:21:26.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:26.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj40 datafile 2021-07-26T06:21:26.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:26.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj41 datafile 2021-07-26T06:21:26.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:26.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj42 datafile 2021-07-26T06:21:26.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:26.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj43 datafile 2021-07-26T06:21:26.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:27.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj44 datafile 2021-07-26T06:21:27.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:27.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj45 datafile 2021-07-26T06:21:27.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:27.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj46 datafile 2021-07-26T06:21:27.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:27.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj47 datafile 2021-07-26T06:21:27.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:27.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj48 datafile 2021-07-26T06:21:27.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:27.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj49 datafile 2021-07-26T06:21:27.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:27.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj50 datafile 2021-07-26T06:21:27.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:27.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj51 datafile 2021-07-26T06:21:28.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:28.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj52 datafile 2021-07-26T06:21:28.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:28.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj53 datafile 2021-07-26T06:21:28.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:28.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj54 datafile 2021-07-26T06:21:28.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:28.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj55 datafile 2021-07-26T06:21:28.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:28.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj56 datafile 2021-07-26T06:21:28.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:28.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj57 datafile 2021-07-26T06:21:28.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:28.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj58 datafile 2021-07-26T06:21:29.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:29.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj59 datafile 2021-07-26T06:21:29.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:29.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj60 datafile 2021-07-26T06:21:29.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:29.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj61 datafile 2021-07-26T06:21:29.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:29.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj62 datafile 2021-07-26T06:21:30.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:30.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj63 datafile 2021-07-26T06:21:30.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:30.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj64 datafile 2021-07-26T06:21:30.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:30.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj65 datafile 2021-07-26T06:21:30.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:30.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj66 datafile 2021-07-26T06:21:30.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:30.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj67 datafile 2021-07-26T06:21:30.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:30.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj68 datafile 2021-07-26T06:21:30.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:30.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj69 datafile 2021-07-26T06:21:30.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:30.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj70 datafile 2021-07-26T06:21:31.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:31.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj71 datafile 2021-07-26T06:21:31.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:31.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj72 datafile 2021-07-26T06:21:31.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:31.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj73 datafile 2021-07-26T06:21:31.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:31.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj74 datafile 2021-07-26T06:21:31.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:31.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj75 datafile 2021-07-26T06:21:31.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:31.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj76 datafile 2021-07-26T06:21:31.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:31.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj77 datafile 2021-07-26T06:21:32.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:32.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj78 datafile 2021-07-26T06:21:32.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:32.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj79 datafile 2021-07-26T06:21:32.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:32.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj80 datafile 2021-07-26T06:21:32.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:32.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj81 datafile 2021-07-26T06:21:32.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:32.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj82 datafile 2021-07-26T06:21:32.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:32.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj83 datafile 2021-07-26T06:21:32.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:32.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj84 datafile 2021-07-26T06:21:33.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:33.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj85 datafile 2021-07-26T06:21:33.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:33.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj86 datafile 2021-07-26T06:21:33.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:33.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj87 datafile 2021-07-26T06:21:33.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:33.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj88 datafile 2021-07-26T06:21:33.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:33.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj89 datafile 2021-07-26T06:21:33.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:33.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj90 datafile 2021-07-26T06:21:33.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:33.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj91 datafile 2021-07-26T06:21:34.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:34.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj92 datafile 2021-07-26T06:21:34.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:34.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj93 datafile 2021-07-26T06:21:34.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:34.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj94 datafile 2021-07-26T06:21:34.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:34.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj95 datafile 2021-07-26T06:21:34.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:34.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj96 datafile 2021-07-26T06:21:34.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:34.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj97 datafile 2021-07-26T06:21:35.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:35.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj98 datafile 2021-07-26T06:21:35.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:35.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj99 datafile 2021-07-26T06:21:35.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:35.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj100 datafile 2021-07-26T06:21:35.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:35.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj101 datafile 2021-07-26T06:21:35.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:35.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj102 datafile 2021-07-26T06:21:35.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:35.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj103 datafile 2021-07-26T06:21:35.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:35.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj104 datafile 2021-07-26T06:21:35.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:35.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj105 datafile 2021-07-26T06:21:35.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:35.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj106 datafile 2021-07-26T06:21:36.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:36.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj107 datafile 2021-07-26T06:21:36.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:36.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj108 datafile 2021-07-26T06:21:36.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:36.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj109 datafile 2021-07-26T06:21:36.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:36.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj110 datafile 2021-07-26T06:21:36.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:36.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj111 datafile 2021-07-26T06:21:37.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:37.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj112 datafile 2021-07-26T06:21:37.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:37.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj113 datafile 2021-07-26T06:21:37.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:37.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj114 datafile 2021-07-26T06:21:37.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:37.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj115 datafile 2021-07-26T06:21:37.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:37.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj116 datafile 2021-07-26T06:21:37.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:37.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj117 datafile 2021-07-26T06:21:37.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:37.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj118 datafile 2021-07-26T06:21:38.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:38.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj119 datafile 2021-07-26T06:21:38.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:38.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj120 datafile 2021-07-26T06:21:38.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:38.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj121 datafile 2021-07-26T06:21:38.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:38.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj122 datafile 2021-07-26T06:21:38.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:38.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj123 datafile 2021-07-26T06:21:39.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:39.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj124 datafile 2021-07-26T06:21:39.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:39.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj125 datafile 2021-07-26T06:21:39.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:39.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj126 datafile 2021-07-26T06:21:39.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:39.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj127 datafile 2021-07-26T06:21:39.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:39.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj128 datafile 2021-07-26T06:21:39.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:39.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj129 datafile 2021-07-26T06:21:39.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:39.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj130 datafile 2021-07-26T06:21:39.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:39.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj131 datafile 2021-07-26T06:21:39.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:39.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj132 datafile 2021-07-26T06:21:39.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:39.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj133 datafile 2021-07-26T06:21:39.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:39.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj134 datafile 2021-07-26T06:21:40.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:40.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj135 datafile 2021-07-26T06:21:40.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:40.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj136 datafile 2021-07-26T06:21:40.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:40.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj137 datafile 2021-07-26T06:21:40.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:40.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj138 datafile 2021-07-26T06:21:40.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:41.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj139 datafile 2021-07-26T06:21:41.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:41.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj140 datafile 2021-07-26T06:21:41.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:41.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj141 datafile 2021-07-26T06:21:41.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:41.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj142 datafile 2021-07-26T06:21:41.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:41.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj143 datafile 2021-07-26T06:21:41.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:41.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj144 datafile 2021-07-26T06:21:41.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:41.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj145 datafile 2021-07-26T06:21:41.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:41.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj146 datafile 2021-07-26T06:21:42.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:42.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj147 datafile 2021-07-26T06:21:42.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:42.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj148 datafile 2021-07-26T06:21:42.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:42.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj149 datafile 2021-07-26T06:21:42.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:42.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj150 datafile 2021-07-26T06:21:42.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:42.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj151 datafile 2021-07-26T06:21:42.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:42.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj152 datafile 2021-07-26T06:21:42.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:42.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj153 datafile 2021-07-26T06:21:42.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:42.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj154 datafile 2021-07-26T06:21:43.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:43.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj155 datafile 2021-07-26T06:21:43.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:43.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj156 datafile 2021-07-26T06:21:43.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:43.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj157 datafile 2021-07-26T06:21:43.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:43.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj158 datafile 2021-07-26T06:21:43.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:43.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj159 datafile 2021-07-26T06:21:43.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:43.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj160 datafile 2021-07-26T06:21:43.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:43.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj161 datafile 2021-07-26T06:21:43.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:43.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj162 datafile 2021-07-26T06:21:44.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:44.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj163 datafile 2021-07-26T06:21:44.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:44.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj164 datafile 2021-07-26T06:21:44.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:44.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj165 datafile 2021-07-26T06:21:44.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:44.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj166 datafile 2021-07-26T06:21:44.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:44.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj167 datafile 2021-07-26T06:21:44.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:44.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj168 datafile 2021-07-26T06:21:45.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:45.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj169 datafile 2021-07-26T06:21:45.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:45.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj170 datafile 2021-07-26T06:21:45.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:45.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj171 datafile 2021-07-26T06:21:45.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:45.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj172 datafile 2021-07-26T06:21:45.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:45.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj173 datafile 2021-07-26T06:21:45.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:45.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj174 datafile 2021-07-26T06:21:45.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:45.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj175 datafile 2021-07-26T06:21:46.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:46.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj176 datafile 2021-07-26T06:21:46.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:46.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj177 datafile 2021-07-26T06:21:46.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:46.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj178 datafile 2021-07-26T06:21:46.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:46.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj179 datafile 2021-07-26T06:21:46.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:46.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj180 datafile 2021-07-26T06:21:46.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:46.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj181 datafile 2021-07-26T06:21:46.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:46.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj182 datafile 2021-07-26T06:21:47.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:47.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj183 datafile 2021-07-26T06:21:47.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:47.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj184 datafile 2021-07-26T06:21:47.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:47.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj185 datafile 2021-07-26T06:21:47.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:47.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj186 datafile 2021-07-26T06:21:47.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:47.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj187 datafile 2021-07-26T06:21:47.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:47.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj188 datafile 2021-07-26T06:21:47.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:47.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj189 datafile 2021-07-26T06:21:47.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:47.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj190 datafile 2021-07-26T06:21:48.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:48.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj191 datafile 2021-07-26T06:21:48.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:48.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj192 datafile 2021-07-26T06:21:48.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:48.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj193 datafile 2021-07-26T06:21:48.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:48.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj194 datafile 2021-07-26T06:21:48.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:48.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj195 datafile 2021-07-26T06:21:48.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:48.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj196 datafile 2021-07-26T06:21:48.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:48.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj197 datafile 2021-07-26T06:21:49.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:49.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj198 datafile 2021-07-26T06:21:49.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:49.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj199 datafile 2021-07-26T06:21:49.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:49.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj200 datafile 2021-07-26T06:21:49.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:49.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj201 datafile 2021-07-26T06:21:49.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:49.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj202 datafile 2021-07-26T06:21:49.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:49.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj203 datafile 2021-07-26T06:21:50.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:50.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj204 datafile 2021-07-26T06:21:50.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:50.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj205 datafile 2021-07-26T06:21:50.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:50.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj206 datafile 2021-07-26T06:21:50.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:50.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj207 datafile 2021-07-26T06:21:50.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:50.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj208 datafile 2021-07-26T06:21:50.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:50.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj209 datafile 2021-07-26T06:21:50.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:50.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj210 datafile 2021-07-26T06:21:50.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:50.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj211 datafile 2021-07-26T06:21:51.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:51.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj212 datafile 2021-07-26T06:21:51.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:51.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj213 datafile 2021-07-26T06:21:51.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:51.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj214 datafile 2021-07-26T06:21:51.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:51.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj215 datafile 2021-07-26T06:21:51.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:51.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj216 datafile 2021-07-26T06:21:51.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:51.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj217 datafile 2021-07-26T06:21:51.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:51.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj218 datafile 2021-07-26T06:21:52.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:52.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj219 datafile 2021-07-26T06:21:52.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:52.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj220 datafile 2021-07-26T06:21:52.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:52.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj221 datafile 2021-07-26T06:21:52.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:52.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj222 datafile 2021-07-26T06:21:52.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:52.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj223 datafile 2021-07-26T06:21:52.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:52.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj224 datafile 2021-07-26T06:21:52.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:52.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj225 datafile 2021-07-26T06:21:53.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:53.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj226 datafile 2021-07-26T06:21:53.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:53.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj227 datafile 2021-07-26T06:21:53.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:53.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj228 datafile 2021-07-26T06:21:53.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:53.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj229 datafile 2021-07-26T06:21:53.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:53.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj230 datafile 2021-07-26T06:21:53.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:53.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj231 datafile 2021-07-26T06:21:53.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:53.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj232 datafile 2021-07-26T06:21:54.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:54.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj233 datafile 2021-07-26T06:21:54.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:54.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj234 datafile 2021-07-26T06:21:54.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:54.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj235 datafile 2021-07-26T06:21:54.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:54.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj236 datafile 2021-07-26T06:21:54.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:54.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj237 datafile 2021-07-26T06:21:54.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:54.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj238 datafile 2021-07-26T06:21:54.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:54.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj239 datafile 2021-07-26T06:21:55.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:55.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj240 datafile 2021-07-26T06:21:55.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:55.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj241 datafile 2021-07-26T06:21:55.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:55.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj242 datafile 2021-07-26T06:21:55.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:55.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj243 datafile 2021-07-26T06:21:55.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:55.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj244 datafile 2021-07-26T06:21:55.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:55.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj245 datafile 2021-07-26T06:21:55.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:55.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj246 datafile 2021-07-26T06:21:56.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:56.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj247 datafile 2021-07-26T06:21:56.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:56.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj248 datafile 2021-07-26T06:21:56.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:56.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj249 datafile 2021-07-26T06:21:56.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:56.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj250 datafile 2021-07-26T06:21:56.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:56.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj251 datafile 2021-07-26T06:21:56.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:56.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj252 datafile 2021-07-26T06:21:56.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:56.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj253 datafile 2021-07-26T06:21:56.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:57.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj254 datafile 2021-07-26T06:21:57.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:57.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj255 datafile 2021-07-26T06:21:57.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:57.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj256 datafile 2021-07-26T06:21:57.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:57.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj257 datafile 2021-07-26T06:21:57.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:57.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj258 datafile 2021-07-26T06:21:57.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:57.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj259 datafile 2021-07-26T06:21:57.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:57.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj260 datafile 2021-07-26T06:21:57.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:57.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj261 datafile 2021-07-26T06:21:58.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:58.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj262 datafile 2021-07-26T06:21:58.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:58.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj263 datafile 2021-07-26T06:21:58.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:58.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj264 datafile 2021-07-26T06:21:58.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:58.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj265 datafile 2021-07-26T06:21:58.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:58.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj266 datafile 2021-07-26T06:21:58.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:58.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj267 datafile 2021-07-26T06:21:58.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:58.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj268 datafile 2021-07-26T06:21:59.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:59.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj269 datafile 2021-07-26T06:21:59.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:59.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj270 datafile 2021-07-26T06:21:59.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:59.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj271 datafile 2021-07-26T06:21:59.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:59.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj272 datafile 2021-07-26T06:21:59.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:59.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj273 datafile 2021-07-26T06:21:59.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:59.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj274 datafile 2021-07-26T06:21:59.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:21:59.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj275 datafile 2021-07-26T06:22:00.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:00.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj276 datafile 2021-07-26T06:22:00.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:00.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj277 datafile 2021-07-26T06:22:00.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:00.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj278 datafile 2021-07-26T06:22:00.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:00.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj279 datafile 2021-07-26T06:22:00.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:00.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj280 datafile 2021-07-26T06:22:00.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:00.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj281 datafile 2021-07-26T06:22:01.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:01.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj282 datafile 2021-07-26T06:22:01.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:01.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj283 datafile 2021-07-26T06:22:01.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:01.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj284 datafile 2021-07-26T06:22:01.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:01.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj285 datafile 2021-07-26T06:22:01.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:01.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj286 datafile 2021-07-26T06:22:02.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:02.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj287 datafile 2021-07-26T06:22:02.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:02.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj288 datafile 2021-07-26T06:22:02.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:02.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj289 datafile 2021-07-26T06:22:02.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:02.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj290 datafile 2021-07-26T06:22:02.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:02.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj291 datafile 2021-07-26T06:22:02.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:02.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj292 datafile 2021-07-26T06:22:02.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:02.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj293 datafile 2021-07-26T06:22:02.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:02.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj294 datafile 2021-07-26T06:22:03.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:03.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj295 datafile 2021-07-26T06:22:03.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:03.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj296 datafile 2021-07-26T06:22:03.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:03.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj297 datafile 2021-07-26T06:22:03.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:03.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj298 datafile 2021-07-26T06:22:03.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:03.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj299 datafile 2021-07-26T06:22:03.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:03.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj300 datafile 2021-07-26T06:22:03.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:03.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj301 datafile 2021-07-26T06:22:04.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:04.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj302 datafile 2021-07-26T06:22:04.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:04.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj303 datafile 2021-07-26T06:22:04.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:04.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj304 datafile 2021-07-26T06:22:04.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:04.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj305 datafile 2021-07-26T06:22:04.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:04.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj306 datafile 2021-07-26T06:22:04.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:04.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj307 datafile 2021-07-26T06:22:05.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:05.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj308 datafile 2021-07-26T06:22:05.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:05.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj309 datafile 2021-07-26T06:22:05.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:05.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj310 datafile 2021-07-26T06:22:05.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:05.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj311 datafile 2021-07-26T06:22:05.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:05.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj312 datafile 2021-07-26T06:22:05.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:05.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj313 datafile 2021-07-26T06:22:05.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:05.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj314 datafile 2021-07-26T06:22:05.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:05.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj315 datafile 2021-07-26T06:22:06.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:06.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj316 datafile 2021-07-26T06:22:06.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:06.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj317 datafile 2021-07-26T06:22:06.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:06.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj318 datafile 2021-07-26T06:22:06.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:06.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj319 datafile 2021-07-26T06:22:06.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:06.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj320 datafile 2021-07-26T06:22:06.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:06.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj321 datafile 2021-07-26T06:22:07.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:07.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj322 datafile 2021-07-26T06:22:07.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:07.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj323 datafile 2021-07-26T06:22:07.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:07.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj324 datafile 2021-07-26T06:22:07.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:07.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj325 datafile 2021-07-26T06:22:07.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:07.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj326 datafile 2021-07-26T06:22:07.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:07.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj327 datafile 2021-07-26T06:22:07.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:07.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj328 datafile 2021-07-26T06:22:07.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:07.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj329 datafile 2021-07-26T06:22:08.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:08.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj330 datafile 2021-07-26T06:22:08.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:08.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj331 datafile 2021-07-26T06:22:08.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:08.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj332 datafile 2021-07-26T06:22:08.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:08.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj333 datafile 2021-07-26T06:22:08.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:08.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj334 datafile 2021-07-26T06:22:08.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:08.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj335 datafile 2021-07-26T06:22:08.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:08.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj336 datafile 2021-07-26T06:22:09.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:09.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj337 datafile 2021-07-26T06:22:09.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:09.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj338 datafile 2021-07-26T06:22:09.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:09.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj339 datafile 2021-07-26T06:22:09.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:09.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj340 datafile 2021-07-26T06:22:09.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:09.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj341 datafile 2021-07-26T06:22:09.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:09.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj342 datafile 2021-07-26T06:22:09.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:09.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj343 datafile 2021-07-26T06:22:10.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:10.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj344 datafile 2021-07-26T06:22:10.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:10.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj345 datafile 2021-07-26T06:22:10.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:10.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj346 datafile 2021-07-26T06:22:10.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:10.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj347 datafile 2021-07-26T06:22:10.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:10.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj348 datafile 2021-07-26T06:22:10.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:10.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj349 datafile 2021-07-26T06:22:10.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:11.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj350 datafile 2021-07-26T06:22:11.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:11.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj351 datafile 2021-07-26T06:22:11.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:11.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj352 datafile 2021-07-26T06:22:11.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:11.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj353 datafile 2021-07-26T06:22:11.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:11.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj354 datafile 2021-07-26T06:22:11.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:11.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj355 datafile 2021-07-26T06:22:11.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:11.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj356 datafile 2021-07-26T06:22:12.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:12.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj357 datafile 2021-07-26T06:22:12.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:12.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj358 datafile 2021-07-26T06:22:12.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:12.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj359 datafile 2021-07-26T06:22:12.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:12.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj360 datafile 2021-07-26T06:22:12.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:12.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj361 datafile 2021-07-26T06:22:12.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:12.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj362 datafile 2021-07-26T06:22:12.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:12.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj363 datafile 2021-07-26T06:22:13.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:13.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj364 datafile 2021-07-26T06:22:13.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:13.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj365 datafile 2021-07-26T06:22:13.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:13.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj366 datafile 2021-07-26T06:22:13.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:13.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj367 datafile 2021-07-26T06:22:13.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:13.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj368 datafile 2021-07-26T06:22:13.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:13.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj369 datafile 2021-07-26T06:22:13.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:13.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj370 datafile 2021-07-26T06:22:14.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:14.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj371 datafile 2021-07-26T06:22:14.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:14.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj372 datafile 2021-07-26T06:22:14.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:14.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj373 datafile 2021-07-26T06:22:14.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:14.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj374 datafile 2021-07-26T06:22:15.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:15.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj375 datafile 2021-07-26T06:22:15.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:15.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj376 datafile 2021-07-26T06:22:15.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:15.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj377 datafile 2021-07-26T06:22:15.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:15.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj378 datafile 2021-07-26T06:22:15.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:15.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj379 datafile 2021-07-26T06:22:15.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:15.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj380 datafile 2021-07-26T06:22:15.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:15.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj381 datafile 2021-07-26T06:22:15.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:15.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj382 datafile 2021-07-26T06:22:15.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:15.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj383 datafile 2021-07-26T06:22:15.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:15.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj384 datafile 2021-07-26T06:22:15.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:15.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj385 datafile 2021-07-26T06:22:16.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:16.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj386 datafile 2021-07-26T06:22:16.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:16.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj387 datafile 2021-07-26T06:22:16.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:16.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj388 datafile 2021-07-26T06:22:16.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:16.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj389 datafile 2021-07-26T06:22:16.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:16.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj390 datafile 2021-07-26T06:22:16.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:16.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj391 datafile 2021-07-26T06:22:17.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:17.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj392 datafile 2021-07-26T06:22:17.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:17.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj393 datafile 2021-07-26T06:22:17.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:17.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj394 datafile 2021-07-26T06:22:17.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:17.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj395 datafile 2021-07-26T06:22:17.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:17.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj396 datafile 2021-07-26T06:22:17.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:17.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj397 datafile 2021-07-26T06:22:17.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:17.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj398 datafile 2021-07-26T06:22:18.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:18.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj399 datafile 2021-07-26T06:22:18.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:18.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj400 datafile 2021-07-26T06:22:18.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:18.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj401 datafile 2021-07-26T06:22:18.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:18.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj402 datafile 2021-07-26T06:22:18.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:18.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj403 datafile 2021-07-26T06:22:18.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:18.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj404 datafile 2021-07-26T06:22:19.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:19.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj405 datafile 2021-07-26T06:22:19.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:19.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj406 datafile 2021-07-26T06:22:19.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:19.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj407 datafile 2021-07-26T06:22:19.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:19.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj408 datafile 2021-07-26T06:22:19.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:19.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj409 datafile 2021-07-26T06:22:19.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:19.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj410 datafile 2021-07-26T06:22:20.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:20.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj411 datafile 2021-07-26T06:22:20.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:20.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj412 datafile 2021-07-26T06:22:20.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:20.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj413 datafile 2021-07-26T06:22:20.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:20.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj414 datafile 2021-07-26T06:22:20.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:20.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj415 datafile 2021-07-26T06:22:20.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:20.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj416 datafile 2021-07-26T06:22:20.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:20.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj417 datafile 2021-07-26T06:22:20.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:20.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj418 datafile 2021-07-26T06:22:20.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:20.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj419 datafile 2021-07-26T06:22:21.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:21.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj420 datafile 2021-07-26T06:22:21.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:21.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj421 datafile 2021-07-26T06:22:21.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:21.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj422 datafile 2021-07-26T06:22:21.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:21.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj423 datafile 2021-07-26T06:22:21.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:21.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj424 datafile 2021-07-26T06:22:21.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:21.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj425 datafile 2021-07-26T06:22:22.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:22.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj426 datafile 2021-07-26T06:22:22.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:22.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj427 datafile 2021-07-26T06:22:22.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:22.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj428 datafile 2021-07-26T06:22:22.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:22.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj429 datafile 2021-07-26T06:22:22.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:22.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj430 datafile 2021-07-26T06:22:22.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:22.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj431 datafile 2021-07-26T06:22:22.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:22.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj432 datafile 2021-07-26T06:22:22.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:22.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj433 datafile 2021-07-26T06:22:23.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:23.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj434 datafile 2021-07-26T06:22:23.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:23.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj435 datafile 2021-07-26T06:22:23.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:23.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj436 datafile 2021-07-26T06:22:23.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:23.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj437 datafile 2021-07-26T06:22:23.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:23.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj438 datafile 2021-07-26T06:22:23.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:23.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj439 datafile 2021-07-26T06:22:23.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:23.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj440 datafile 2021-07-26T06:22:24.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:24.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj441 datafile 2021-07-26T06:22:24.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:24.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj442 datafile 2021-07-26T06:22:24.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:24.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj443 datafile 2021-07-26T06:22:24.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:24.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj444 datafile 2021-07-26T06:22:24.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:24.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj445 datafile 2021-07-26T06:22:24.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:24.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj446 datafile 2021-07-26T06:22:24.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:24.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj447 datafile 2021-07-26T06:22:25.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:25.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj448 datafile 2021-07-26T06:22:25.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:25.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj449 datafile 2021-07-26T06:22:25.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:25.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj450 datafile 2021-07-26T06:22:25.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:25.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj451 datafile 2021-07-26T06:22:25.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:25.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj452 datafile 2021-07-26T06:22:25.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:25.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj453 datafile 2021-07-26T06:22:25.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:25.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj454 datafile 2021-07-26T06:22:26.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:26.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj455 datafile 2021-07-26T06:22:26.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:26.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj456 datafile 2021-07-26T06:22:26.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:26.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj457 datafile 2021-07-26T06:22:26.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:26.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj458 datafile 2021-07-26T06:22:26.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:26.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj459 datafile 2021-07-26T06:22:26.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:26.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj460 datafile 2021-07-26T06:22:26.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:26.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj461 datafile 2021-07-26T06:22:27.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:27.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj462 datafile 2021-07-26T06:22:27.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:27.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj463 datafile 2021-07-26T06:22:27.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:27.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj464 datafile 2021-07-26T06:22:27.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:27.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj465 datafile 2021-07-26T06:22:27.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:27.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj466 datafile 2021-07-26T06:22:27.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:27.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj467 datafile 2021-07-26T06:22:27.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:27.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj468 datafile 2021-07-26T06:22:27.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:27.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj469 datafile 2021-07-26T06:22:28.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:28.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj470 datafile 2021-07-26T06:22:28.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:28.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj471 datafile 2021-07-26T06:22:28.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:28.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj472 datafile 2021-07-26T06:22:28.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:28.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj473 datafile 2021-07-26T06:22:28.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:28.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj474 datafile 2021-07-26T06:22:28.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:28.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj475 datafile 2021-07-26T06:22:29.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:29.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj476 datafile 2021-07-26T06:22:29.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:29.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj477 datafile 2021-07-26T06:22:29.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:29.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj478 datafile 2021-07-26T06:22:29.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:29.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj479 datafile 2021-07-26T06:22:29.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:29.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj480 datafile 2021-07-26T06:22:29.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:29.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj481 datafile 2021-07-26T06:22:29.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:29.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj482 datafile 2021-07-26T06:22:30.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:30.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj483 datafile 2021-07-26T06:22:30.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:30.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj484 datafile 2021-07-26T06:22:30.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:30.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj485 datafile 2021-07-26T06:22:30.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:30.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj486 datafile 2021-07-26T06:22:30.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:30.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj487 datafile 2021-07-26T06:22:30.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:30.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj488 datafile 2021-07-26T06:22:30.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:30.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj489 datafile 2021-07-26T06:22:31.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:31.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj490 datafile 2021-07-26T06:22:31.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:31.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj491 datafile 2021-07-26T06:22:31.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:31.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj492 datafile 2021-07-26T06:22:31.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:31.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj493 datafile 2021-07-26T06:22:31.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:31.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj494 datafile 2021-07-26T06:22:31.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:31.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj495 datafile 2021-07-26T06:22:31.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:31.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj496 datafile 2021-07-26T06:22:32.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:32.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj497 datafile 2021-07-26T06:22:32.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:32.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj498 datafile 2021-07-26T06:22:32.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:32.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj499 datafile 2021-07-26T06:22:32.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:32.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj500 datafile 2021-07-26T06:22:32.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:32.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj501 datafile 2021-07-26T06:22:33.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:33.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj502 datafile 2021-07-26T06:22:33.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:33.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj503 datafile 2021-07-26T06:22:33.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:33.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj504 datafile 2021-07-26T06:22:33.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:33.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj505 datafile 2021-07-26T06:22:33.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:33.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj506 datafile 2021-07-26T06:22:33.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:33.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj507 datafile 2021-07-26T06:22:33.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:33.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj508 datafile 2021-07-26T06:22:34.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:34.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj509 datafile 2021-07-26T06:22:34.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:34.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj510 datafile 2021-07-26T06:22:34.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:34.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj511 datafile 2021-07-26T06:22:34.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:34.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj512 datafile 2021-07-26T06:22:34.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:34.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj513 datafile 2021-07-26T06:22:34.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:34.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj514 datafile 2021-07-26T06:22:34.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:34.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj515 datafile 2021-07-26T06:22:35.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:35.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj516 datafile 2021-07-26T06:22:35.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:35.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj517 datafile 2021-07-26T06:22:35.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:35.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj518 datafile 2021-07-26T06:22:35.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:35.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj519 datafile 2021-07-26T06:22:35.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:35.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj520 datafile 2021-07-26T06:22:35.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:35.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj521 datafile 2021-07-26T06:22:35.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:35.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj522 datafile 2021-07-26T06:22:36.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:36.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj523 datafile 2021-07-26T06:22:36.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:36.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj524 datafile 2021-07-26T06:22:36.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:36.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj525 datafile 2021-07-26T06:22:36.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:36.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj526 datafile 2021-07-26T06:22:36.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:36.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj527 datafile 2021-07-26T06:22:36.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:36.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj528 datafile 2021-07-26T06:22:36.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:36.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj529 datafile 2021-07-26T06:22:37.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:37.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj530 datafile 2021-07-26T06:22:37.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:37.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj531 datafile 2021-07-26T06:22:37.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:37.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj532 datafile 2021-07-26T06:22:37.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:37.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj533 datafile 2021-07-26T06:22:37.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:37.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj534 datafile 2021-07-26T06:22:37.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:37.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj535 datafile 2021-07-26T06:22:37.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:37.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj536 datafile 2021-07-26T06:22:38.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:38.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj537 datafile 2021-07-26T06:22:38.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:38.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj538 datafile 2021-07-26T06:22:38.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:38.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj539 datafile 2021-07-26T06:22:38.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:38.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj540 datafile 2021-07-26T06:22:38.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:38.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj541 datafile 2021-07-26T06:22:38.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:38.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj542 datafile 2021-07-26T06:22:38.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:38.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj543 datafile 2021-07-26T06:22:38.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:38.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj544 datafile 2021-07-26T06:22:39.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:39.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj545 datafile 2021-07-26T06:22:39.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:39.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj546 datafile 2021-07-26T06:22:39.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:39.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj547 datafile 2021-07-26T06:22:39.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:39.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj548 datafile 2021-07-26T06:22:39.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:39.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj549 datafile 2021-07-26T06:22:39.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:39.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj550 datafile 2021-07-26T06:22:39.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:39.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj551 datafile 2021-07-26T06:22:40.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:40.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj552 datafile 2021-07-26T06:22:40.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:40.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj553 datafile 2021-07-26T06:22:40.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:40.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj554 datafile 2021-07-26T06:22:40.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:40.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj555 datafile 2021-07-26T06:22:40.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:40.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj556 datafile 2021-07-26T06:22:40.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:40.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj557 datafile 2021-07-26T06:22:40.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:40.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj558 datafile 2021-07-26T06:22:41.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:41.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj559 datafile 2021-07-26T06:22:41.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:41.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj560 datafile 2021-07-26T06:22:41.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:41.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj561 datafile 2021-07-26T06:22:41.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:41.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj562 datafile 2021-07-26T06:22:41.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:41.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj563 datafile 2021-07-26T06:22:41.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:41.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj564 datafile 2021-07-26T06:22:41.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:41.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj565 datafile 2021-07-26T06:22:42.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:42.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj566 datafile 2021-07-26T06:22:42.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:42.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj567 datafile 2021-07-26T06:22:42.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:42.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj568 datafile 2021-07-26T06:22:42.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:42.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj569 datafile 2021-07-26T06:22:42.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:42.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj570 datafile 2021-07-26T06:22:42.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:42.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj571 datafile 2021-07-26T06:22:43.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:43.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj572 datafile 2021-07-26T06:22:43.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:43.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj573 datafile 2021-07-26T06:22:43.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:43.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj574 datafile 2021-07-26T06:22:43.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:43.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj575 datafile 2021-07-26T06:22:43.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:43.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj576 datafile 2021-07-26T06:22:43.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:43.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj577 datafile 2021-07-26T06:22:44.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:44.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj578 datafile 2021-07-26T06:22:44.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:44.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj579 datafile 2021-07-26T06:22:44.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:44.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj580 datafile 2021-07-26T06:22:44.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:44.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj581 datafile 2021-07-26T06:22:44.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:44.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj582 datafile 2021-07-26T06:22:44.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:44.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj583 datafile 2021-07-26T06:22:44.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:44.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj584 datafile 2021-07-26T06:22:44.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:44.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj585 datafile 2021-07-26T06:22:45.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:45.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj586 datafile 2021-07-26T06:22:45.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:45.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj587 datafile 2021-07-26T06:22:45.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:45.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj588 datafile 2021-07-26T06:22:45.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:45.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj589 datafile 2021-07-26T06:22:45.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:45.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj590 datafile 2021-07-26T06:22:45.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:45.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj591 datafile 2021-07-26T06:22:45.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:45.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj592 datafile 2021-07-26T06:22:46.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:46.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj593 datafile 2021-07-26T06:22:46.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:46.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj594 datafile 2021-07-26T06:22:46.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:46.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj595 datafile 2021-07-26T06:22:46.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:46.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj596 datafile 2021-07-26T06:22:46.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:46.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj597 datafile 2021-07-26T06:22:46.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:46.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj598 datafile 2021-07-26T06:22:47.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:47.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj599 datafile 2021-07-26T06:22:47.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:47.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj600 datafile 2021-07-26T06:22:47.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:47.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj601 datafile 2021-07-26T06:22:47.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:47.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj602 datafile 2021-07-26T06:22:47.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:47.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj603 datafile 2021-07-26T06:22:47.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:47.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj604 datafile 2021-07-26T06:22:48.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:48.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj605 datafile 2021-07-26T06:22:48.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:48.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj606 datafile 2021-07-26T06:22:48.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:48.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj607 datafile 2021-07-26T06:22:48.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:48.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj608 datafile 2021-07-26T06:22:48.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:48.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj609 datafile 2021-07-26T06:22:48.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:48.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj610 datafile 2021-07-26T06:22:48.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:48.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj611 datafile 2021-07-26T06:22:49.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:49.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj612 datafile 2021-07-26T06:22:49.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:49.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj613 datafile 2021-07-26T06:22:49.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:49.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj614 datafile 2021-07-26T06:22:49.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:49.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj615 datafile 2021-07-26T06:22:49.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:49.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj616 datafile 2021-07-26T06:22:49.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:49.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj617 datafile 2021-07-26T06:22:49.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:49.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj618 datafile 2021-07-26T06:22:50.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:50.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj619 datafile 2021-07-26T06:22:50.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:50.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj620 datafile 2021-07-26T06:22:50.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:50.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj621 datafile 2021-07-26T06:22:50.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:50.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj622 datafile 2021-07-26T06:22:50.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:50.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj623 datafile 2021-07-26T06:22:50.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:50.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj624 datafile 2021-07-26T06:22:50.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:50.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj625 datafile 2021-07-26T06:22:51.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:51.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj626 datafile 2021-07-26T06:22:51.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:51.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj627 datafile 2021-07-26T06:22:51.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:51.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj628 datafile 2021-07-26T06:22:51.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:51.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj629 datafile 2021-07-26T06:22:51.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:51.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj630 datafile 2021-07-26T06:22:51.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:51.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj631 datafile 2021-07-26T06:22:52.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:52.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj632 datafile 2021-07-26T06:22:52.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:52.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj633 datafile 2021-07-26T06:22:52.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:52.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj634 datafile 2021-07-26T06:22:52.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:52.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj635 datafile 2021-07-26T06:22:52.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:52.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj636 datafile 2021-07-26T06:22:52.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:52.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj637 datafile 2021-07-26T06:22:52.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:52.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj638 datafile 2021-07-26T06:22:52.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:52.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj639 datafile 2021-07-26T06:22:53.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:53.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj640 datafile 2021-07-26T06:22:53.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:53.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj641 datafile 2021-07-26T06:22:53.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:53.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj642 datafile 2021-07-26T06:22:53.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:53.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj643 datafile 2021-07-26T06:22:53.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:53.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj644 datafile 2021-07-26T06:22:53.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:53.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj645 datafile 2021-07-26T06:22:53.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:53.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj646 datafile 2021-07-26T06:22:54.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:54.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj647 datafile 2021-07-26T06:22:54.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:54.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj648 datafile 2021-07-26T06:22:54.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:54.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj649 datafile 2021-07-26T06:22:54.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:54.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj650 datafile 2021-07-26T06:22:54.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:54.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj651 datafile 2021-07-26T06:22:54.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:54.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj652 datafile 2021-07-26T06:22:54.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:54.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj653 datafile 2021-07-26T06:22:55.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:55.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj654 datafile 2021-07-26T06:22:55.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:55.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj655 datafile 2021-07-26T06:22:55.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:55.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj656 datafile 2021-07-26T06:22:55.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:55.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj657 datafile 2021-07-26T06:22:55.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:55.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj658 datafile 2021-07-26T06:22:55.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:55.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj659 datafile 2021-07-26T06:22:55.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:55.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj660 datafile 2021-07-26T06:22:55.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:56.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj661 datafile 2021-07-26T06:22:56.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:56.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj662 datafile 2021-07-26T06:22:56.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:56.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj663 datafile 2021-07-26T06:22:56.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:56.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj664 datafile 2021-07-26T06:22:56.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:56.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj665 datafile 2021-07-26T06:22:56.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:56.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj666 datafile 2021-07-26T06:22:56.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:56.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj667 datafile 2021-07-26T06:22:57.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:57.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj668 datafile 2021-07-26T06:22:57.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:57.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj669 datafile 2021-07-26T06:22:57.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:57.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj670 datafile 2021-07-26T06:22:57.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:57.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj671 datafile 2021-07-26T06:22:57.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:57.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj672 datafile 2021-07-26T06:22:57.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:57.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj673 datafile 2021-07-26T06:22:57.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:57.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj674 datafile 2021-07-26T06:22:58.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:58.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj675 datafile 2021-07-26T06:22:58.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:58.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj676 datafile 2021-07-26T06:22:58.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:58.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj677 datafile 2021-07-26T06:22:58.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:58.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj678 datafile 2021-07-26T06:22:58.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:58.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj679 datafile 2021-07-26T06:22:58.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:58.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj680 datafile 2021-07-26T06:22:58.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:58.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj681 datafile 2021-07-26T06:22:59.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:59.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj682 datafile 2021-07-26T06:22:59.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:59.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj683 datafile 2021-07-26T06:22:59.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:59.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj684 datafile 2021-07-26T06:22:59.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:59.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj685 datafile 2021-07-26T06:22:59.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:59.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj686 datafile 2021-07-26T06:22:59.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:59.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj687 datafile 2021-07-26T06:22:59.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:22:59.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj688 datafile 2021-07-26T06:23:00.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:00.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj689 datafile 2021-07-26T06:23:00.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:00.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj690 datafile 2021-07-26T06:23:00.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:00.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj691 datafile 2021-07-26T06:23:00.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:00.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj692 datafile 2021-07-26T06:23:00.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:00.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj693 datafile 2021-07-26T06:23:00.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:00.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj694 datafile 2021-07-26T06:23:00.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:00.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj695 datafile 2021-07-26T06:23:00.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:00.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj696 datafile 2021-07-26T06:23:01.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:01.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj697 datafile 2021-07-26T06:23:01.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:01.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj698 datafile 2021-07-26T06:23:01.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:01.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj699 datafile 2021-07-26T06:23:01.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:01.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj700 datafile 2021-07-26T06:23:01.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:01.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj701 datafile 2021-07-26T06:23:01.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:01.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj702 datafile 2021-07-26T06:23:01.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:01.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj703 datafile 2021-07-26T06:23:02.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:02.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj704 datafile 2021-07-26T06:23:02.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:02.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj705 datafile 2021-07-26T06:23:02.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:02.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj706 datafile 2021-07-26T06:23:02.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:02.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj707 datafile 2021-07-26T06:23:02.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:02.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj708 datafile 2021-07-26T06:23:02.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:02.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj709 datafile 2021-07-26T06:23:02.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:02.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj710 datafile 2021-07-26T06:23:03.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:03.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj711 datafile 2021-07-26T06:23:03.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:03.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj712 datafile 2021-07-26T06:23:03.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:03.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj713 datafile 2021-07-26T06:23:03.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:03.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj714 datafile 2021-07-26T06:23:03.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:03.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj715 datafile 2021-07-26T06:23:04.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:04.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj716 datafile 2021-07-26T06:23:04.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:04.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj717 datafile 2021-07-26T06:23:04.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:04.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj718 datafile 2021-07-26T06:23:04.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:04.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj719 datafile 2021-07-26T06:23:04.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:04.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj720 datafile 2021-07-26T06:23:04.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:04.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj721 datafile 2021-07-26T06:23:04.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:04.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj722 datafile 2021-07-26T06:23:04.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:04.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj723 datafile 2021-07-26T06:23:05.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:05.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj724 datafile 2021-07-26T06:23:05.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:05.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj725 datafile 2021-07-26T06:23:05.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:05.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj726 datafile 2021-07-26T06:23:05.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:05.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj727 datafile 2021-07-26T06:23:05.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:05.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj728 datafile 2021-07-26T06:23:05.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:05.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj729 datafile 2021-07-26T06:23:05.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:05.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj730 datafile 2021-07-26T06:23:06.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:06.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj731 datafile 2021-07-26T06:23:06.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:06.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj732 datafile 2021-07-26T06:23:06.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:06.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj733 datafile 2021-07-26T06:23:06.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:06.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj734 datafile 2021-07-26T06:23:06.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:06.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj735 datafile 2021-07-26T06:23:06.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:06.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj736 datafile 2021-07-26T06:23:06.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:06.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj737 datafile 2021-07-26T06:23:07.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:07.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj738 datafile 2021-07-26T06:23:07.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:07.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj739 datafile 2021-07-26T06:23:07.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:07.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj740 datafile 2021-07-26T06:23:07.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:07.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj741 datafile 2021-07-26T06:23:07.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:07.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj742 datafile 2021-07-26T06:23:07.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:07.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj743 datafile 2021-07-26T06:23:08.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:08.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj744 datafile 2021-07-26T06:23:08.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:08.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj745 datafile 2021-07-26T06:23:08.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:08.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj746 datafile 2021-07-26T06:23:08.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:08.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj747 datafile 2021-07-26T06:23:08.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:08.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj748 datafile 2021-07-26T06:23:08.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:08.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj749 datafile 2021-07-26T06:23:08.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:08.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj750 datafile 2021-07-26T06:23:08.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:08.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj751 datafile 2021-07-26T06:23:09.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:09.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj752 datafile 2021-07-26T06:23:09.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:09.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj753 datafile 2021-07-26T06:23:09.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:09.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj754 datafile 2021-07-26T06:23:09.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:09.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj755 datafile 2021-07-26T06:23:09.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:09.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj756 datafile 2021-07-26T06:23:09.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:09.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj757 datafile 2021-07-26T06:23:09.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:09.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj758 datafile 2021-07-26T06:23:10.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:10.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj759 datafile 2021-07-26T06:23:10.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:10.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj760 datafile 2021-07-26T06:23:10.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:10.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj761 datafile 2021-07-26T06:23:10.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:10.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj762 datafile 2021-07-26T06:23:10.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:10.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj763 datafile 2021-07-26T06:23:10.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:10.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj764 datafile 2021-07-26T06:23:10.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:10.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj765 datafile 2021-07-26T06:23:11.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:11.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj766 datafile 2021-07-26T06:23:11.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:11.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj767 datafile 2021-07-26T06:23:11.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:11.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj768 datafile 2021-07-26T06:23:11.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:11.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj769 datafile 2021-07-26T06:23:11.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:11.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj770 datafile 2021-07-26T06:23:11.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:11.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj771 datafile 2021-07-26T06:23:11.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:11.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj772 datafile 2021-07-26T06:23:12.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:12.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj773 datafile 2021-07-26T06:23:12.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:12.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj774 datafile 2021-07-26T06:23:12.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:12.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj775 datafile 2021-07-26T06:23:12.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:12.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj776 datafile 2021-07-26T06:23:12.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:12.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj777 datafile 2021-07-26T06:23:12.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:12.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj778 datafile 2021-07-26T06:23:12.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:12.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj779 datafile 2021-07-26T06:23:13.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:13.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj780 datafile 2021-07-26T06:23:13.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:13.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj781 datafile 2021-07-26T06:23:13.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:13.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj782 datafile 2021-07-26T06:23:13.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:13.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj783 datafile 2021-07-26T06:23:13.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:13.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj784 datafile 2021-07-26T06:23:13.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:13.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj785 datafile 2021-07-26T06:23:13.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:13.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj786 datafile 2021-07-26T06:23:14.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:14.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj787 datafile 2021-07-26T06:23:14.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:14.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj788 datafile 2021-07-26T06:23:14.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:14.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj789 datafile 2021-07-26T06:23:14.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:14.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj790 datafile 2021-07-26T06:23:14.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:14.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj791 datafile 2021-07-26T06:23:14.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:14.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj792 datafile 2021-07-26T06:23:15.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:15.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj793 datafile 2021-07-26T06:23:15.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:15.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj794 datafile 2021-07-26T06:23:15.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:15.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj795 datafile 2021-07-26T06:23:15.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:15.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj796 datafile 2021-07-26T06:23:15.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:15.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj797 datafile 2021-07-26T06:23:15.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:15.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj798 datafile 2021-07-26T06:23:15.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:15.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj799 datafile 2021-07-26T06:23:16.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:16.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj800 datafile 2021-07-26T06:23:16.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:16.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj801 datafile 2021-07-26T06:23:16.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:16.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj802 datafile 2021-07-26T06:23:16.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:16.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj803 datafile 2021-07-26T06:23:16.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:16.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj804 datafile 2021-07-26T06:23:16.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:16.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj805 datafile 2021-07-26T06:23:16.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:16.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj806 datafile 2021-07-26T06:23:17.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:17.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj807 datafile 2021-07-26T06:23:17.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:17.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj808 datafile 2021-07-26T06:23:17.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:17.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj809 datafile 2021-07-26T06:23:17.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:17.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj810 datafile 2021-07-26T06:23:17.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:17.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj811 datafile 2021-07-26T06:23:17.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:17.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj812 datafile 2021-07-26T06:23:17.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:17.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj813 datafile 2021-07-26T06:23:18.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:18.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj814 datafile 2021-07-26T06:23:18.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:18.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj815 datafile 2021-07-26T06:23:18.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:18.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj816 datafile 2021-07-26T06:23:18.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:18.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj817 datafile 2021-07-26T06:23:18.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:18.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj818 datafile 2021-07-26T06:23:18.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:18.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj819 datafile 2021-07-26T06:23:18.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:18.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj820 datafile 2021-07-26T06:23:19.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:19.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj821 datafile 2021-07-26T06:23:19.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:19.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj822 datafile 2021-07-26T06:23:19.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:19.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj823 datafile 2021-07-26T06:23:19.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:19.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj824 datafile 2021-07-26T06:23:19.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:19.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj825 datafile 2021-07-26T06:23:19.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:19.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj826 datafile 2021-07-26T06:23:20.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:20.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj827 datafile 2021-07-26T06:23:20.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:20.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj828 datafile 2021-07-26T06:23:20.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:20.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj829 datafile 2021-07-26T06:23:20.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:20.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj830 datafile 2021-07-26T06:23:20.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:20.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj831 datafile 2021-07-26T06:23:20.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:20.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj832 datafile 2021-07-26T06:23:20.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:20.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj833 datafile 2021-07-26T06:23:21.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:21.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj834 datafile 2021-07-26T06:23:21.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:21.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj835 datafile 2021-07-26T06:23:21.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:21.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj836 datafile 2021-07-26T06:23:21.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:21.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj837 datafile 2021-07-26T06:23:21.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:21.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj838 datafile 2021-07-26T06:23:21.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:21.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj839 datafile 2021-07-26T06:23:21.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:21.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj840 datafile 2021-07-26T06:23:22.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:22.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj841 datafile 2021-07-26T06:23:22.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:22.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj842 datafile 2021-07-26T06:23:22.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:22.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj843 datafile 2021-07-26T06:23:22.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:22.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj844 datafile 2021-07-26T06:23:22.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:22.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj845 datafile 2021-07-26T06:23:22.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:22.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj846 datafile 2021-07-26T06:23:22.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:22.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj847 datafile 2021-07-26T06:23:23.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:23.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj848 datafile 2021-07-26T06:23:23.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:23.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj849 datafile 2021-07-26T06:23:23.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:23.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj850 datafile 2021-07-26T06:23:23.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:23.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj851 datafile 2021-07-26T06:23:23.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:23.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj852 datafile 2021-07-26T06:23:23.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:23.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj853 datafile 2021-07-26T06:23:23.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:23.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj854 datafile 2021-07-26T06:23:24.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:24.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj855 datafile 2021-07-26T06:23:24.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:24.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj856 datafile 2021-07-26T06:23:24.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:24.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj857 datafile 2021-07-26T06:23:24.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:24.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj858 datafile 2021-07-26T06:23:24.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:24.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj859 datafile 2021-07-26T06:23:24.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:24.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj860 datafile 2021-07-26T06:23:24.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:24.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj861 datafile 2021-07-26T06:23:25.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:25.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj862 datafile 2021-07-26T06:23:25.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:25.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj863 datafile 2021-07-26T06:23:25.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:25.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj864 datafile 2021-07-26T06:23:25.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:25.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj865 datafile 2021-07-26T06:23:25.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:25.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj866 datafile 2021-07-26T06:23:25.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:25.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj867 datafile 2021-07-26T06:23:25.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:25.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj868 datafile 2021-07-26T06:23:26.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:26.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj869 datafile 2021-07-26T06:23:26.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:26.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj870 datafile 2021-07-26T06:23:26.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:26.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj871 datafile 2021-07-26T06:23:26.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:26.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj872 datafile 2021-07-26T06:23:26.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:26.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj873 datafile 2021-07-26T06:23:26.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:26.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj874 datafile 2021-07-26T06:23:26.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:26.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj875 datafile 2021-07-26T06:23:26.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:27.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj876 datafile 2021-07-26T06:23:27.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:27.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj877 datafile 2021-07-26T06:23:27.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:27.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj878 datafile 2021-07-26T06:23:27.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:27.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj879 datafile 2021-07-26T06:23:27.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:27.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj880 datafile 2021-07-26T06:23:27.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:27.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj881 datafile 2021-07-26T06:23:27.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:27.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj882 datafile 2021-07-26T06:23:28.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:28.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj883 datafile 2021-07-26T06:23:28.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:28.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj884 datafile 2021-07-26T06:23:28.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:28.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj885 datafile 2021-07-26T06:23:28.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:28.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj886 datafile 2021-07-26T06:23:28.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:28.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj887 datafile 2021-07-26T06:23:28.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:28.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj888 datafile 2021-07-26T06:23:28.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:28.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj889 datafile 2021-07-26T06:23:28.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:28.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj890 datafile 2021-07-26T06:23:29.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:29.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj891 datafile 2021-07-26T06:23:29.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:29.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj892 datafile 2021-07-26T06:23:29.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:29.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj893 datafile 2021-07-26T06:23:29.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:29.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj894 datafile 2021-07-26T06:23:29.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:29.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj895 datafile 2021-07-26T06:23:29.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:29.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj896 datafile 2021-07-26T06:23:29.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:29.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj897 datafile 2021-07-26T06:23:30.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:30.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj898 datafile 2021-07-26T06:23:30.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:30.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj899 datafile 2021-07-26T06:23:30.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:30.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj900 datafile 2021-07-26T06:23:30.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:30.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj901 datafile 2021-07-26T06:23:30.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:30.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj902 datafile 2021-07-26T06:23:30.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:30.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj903 datafile 2021-07-26T06:23:30.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:30.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj904 datafile 2021-07-26T06:23:31.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:31.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj905 datafile 2021-07-26T06:23:31.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:31.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj906 datafile 2021-07-26T06:23:31.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:31.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj907 datafile 2021-07-26T06:23:31.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:31.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj908 datafile 2021-07-26T06:23:31.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:31.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj909 datafile 2021-07-26T06:23:31.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:31.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj910 datafile 2021-07-26T06:23:31.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:31.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj911 datafile 2021-07-26T06:23:32.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:32.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj912 datafile 2021-07-26T06:23:32.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:32.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj913 datafile 2021-07-26T06:23:32.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:32.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj914 datafile 2021-07-26T06:23:32.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:32.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj915 datafile 2021-07-26T06:23:32.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:32.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj916 datafile 2021-07-26T06:23:32.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:32.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj917 datafile 2021-07-26T06:23:32.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:32.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj918 datafile 2021-07-26T06:23:33.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:33.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj919 datafile 2021-07-26T06:23:33.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:33.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj920 datafile 2021-07-26T06:23:33.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:33.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj921 datafile 2021-07-26T06:23:33.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:33.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj922 datafile 2021-07-26T06:23:33.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:33.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj923 datafile 2021-07-26T06:23:33.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:33.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj924 datafile 2021-07-26T06:23:33.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:33.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj925 datafile 2021-07-26T06:23:34.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:34.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj926 datafile 2021-07-26T06:23:34.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:34.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj927 datafile 2021-07-26T06:23:34.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:34.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj928 datafile 2021-07-26T06:23:34.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:34.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj929 datafile 2021-07-26T06:23:34.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:34.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj930 datafile 2021-07-26T06:23:34.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:34.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj931 datafile 2021-07-26T06:23:35.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:35.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj932 datafile 2021-07-26T06:23:35.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:35.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj933 datafile 2021-07-26T06:23:35.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:35.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj934 datafile 2021-07-26T06:23:35.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:35.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj935 datafile 2021-07-26T06:23:35.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:35.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj936 datafile 2021-07-26T06:23:35.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:35.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj937 datafile 2021-07-26T06:23:35.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:35.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj938 datafile 2021-07-26T06:23:36.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:36.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj939 datafile 2021-07-26T06:23:36.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:36.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj940 datafile 2021-07-26T06:23:36.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:36.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj941 datafile 2021-07-26T06:23:36.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:36.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj942 datafile 2021-07-26T06:23:36.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:36.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj943 datafile 2021-07-26T06:23:36.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:36.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj944 datafile 2021-07-26T06:23:36.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:36.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj945 datafile 2021-07-26T06:23:37.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:37.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj946 datafile 2021-07-26T06:23:37.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:37.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj947 datafile 2021-07-26T06:23:37.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:37.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj948 datafile 2021-07-26T06:23:37.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:37.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj949 datafile 2021-07-26T06:23:37.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:37.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj950 datafile 2021-07-26T06:23:37.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:37.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj951 datafile 2021-07-26T06:23:38.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:38.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj952 datafile 2021-07-26T06:23:38.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:38.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj953 datafile 2021-07-26T06:23:38.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:38.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj954 datafile 2021-07-26T06:23:38.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:38.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj955 datafile 2021-07-26T06:23:38.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:38.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj956 datafile 2021-07-26T06:23:38.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:38.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj957 datafile 2021-07-26T06:23:38.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:38.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj958 datafile 2021-07-26T06:23:38.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:38.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj959 datafile 2021-07-26T06:23:39.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:39.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj960 datafile 2021-07-26T06:23:39.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:39.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj961 datafile 2021-07-26T06:23:39.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:39.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj962 datafile 2021-07-26T06:23:39.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:39.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj963 datafile 2021-07-26T06:23:39.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:39.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj964 datafile 2021-07-26T06:23:39.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:39.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj965 datafile 2021-07-26T06:23:40.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:40.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj966 datafile 2021-07-26T06:23:40.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:40.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj967 datafile 2021-07-26T06:23:40.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:40.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj968 datafile 2021-07-26T06:23:40.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:40.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj969 datafile 2021-07-26T06:23:40.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:40.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj970 datafile 2021-07-26T06:23:40.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:40.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj971 datafile 2021-07-26T06:23:40.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:40.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj972 datafile 2021-07-26T06:23:40.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:40.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj973 datafile 2021-07-26T06:23:41.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:41.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj974 datafile 2021-07-26T06:23:41.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:41.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj975 datafile 2021-07-26T06:23:41.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:41.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj976 datafile 2021-07-26T06:23:41.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:41.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj977 datafile 2021-07-26T06:23:41.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:41.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj978 datafile 2021-07-26T06:23:41.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:41.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj979 datafile 2021-07-26T06:23:42.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:42.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj980 datafile 2021-07-26T06:23:42.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:42.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj981 datafile 2021-07-26T06:23:42.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:42.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj982 datafile 2021-07-26T06:23:42.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:42.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj983 datafile 2021-07-26T06:23:42.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:42.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj984 datafile 2021-07-26T06:23:42.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:42.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj985 datafile 2021-07-26T06:23:42.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:42.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj986 datafile 2021-07-26T06:23:43.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:43.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj987 datafile 2021-07-26T06:23:43.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:43.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj988 datafile 2021-07-26T06:23:43.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:43.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj989 datafile 2021-07-26T06:23:43.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:43.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj990 datafile 2021-07-26T06:23:43.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:43.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj991 datafile 2021-07-26T06:23:43.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:43.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj992 datafile 2021-07-26T06:23:43.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:43.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj993 datafile 2021-07-26T06:23:44.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:44.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj994 datafile 2021-07-26T06:23:44.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:44.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj995 datafile 2021-07-26T06:23:44.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:44.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj996 datafile 2021-07-26T06:23:44.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:44.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj997 datafile 2021-07-26T06:23:44.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:44.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj998 datafile 2021-07-26T06:23:44.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:44.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj999 datafile 2021-07-26T06:23:44.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2021-07-26T06:23:44.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj1000 datafile 2021-07-26T06:23:44.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:79: TEST_recover_unexpected: rm datafile 2021-07-26T06:23:44.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:81: TEST_recover_unexpected: ceph osd set noscrub 2021-07-26T06:23:45.526 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:23:45.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:82: TEST_recover_unexpected: ceph osd set nodeep-scrub 2021-07-26T06:23:46.163 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:23:46.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: ceph pg dump pgs --format=json-pretty 2021-07-26T06:23:46.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: jq '.pg_stats[].pgid' 2021-07-26T06:23:46.446 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:46.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:46.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:46.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.d") | .acting_primary' 2021-07-26T06:23:46.724 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:46.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2021-07-26T06:23:46.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.d"' 2021-07-26T06:23:46.730 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.d 2021-07-26T06:23:46.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.d scrub 2021-07-26T06:23:46.881 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:46.881 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:46.882 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:06.851154+0000" 2021-07-26T06:23:46.882 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:46.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:46.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:46.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.f") | .acting_primary' 2021-07-26T06:23:47.132 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:47.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2021-07-26T06:23:47.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.f"' 2021-07-26T06:23:47.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.f 2021-07-26T06:23:47.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.f scrub 2021-07-26T06:23:47.256 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:47.256 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:47.256 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:07.255765+0000" 2021-07-26T06:23:47.257 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:47.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:47.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:47.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.c") | .acting_primary' 2021-07-26T06:23:47.575 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:47.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=2 2021-07-26T06:23:47.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.c"' 2021-07-26T06:23:47.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.c 2021-07-26T06:23:47.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.c scrub 2021-07-26T06:23:47.681 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:47.682 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:47.682 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:07.681429+0000" 2021-07-26T06:23:47.682 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:47.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:47.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:47.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.c") | .acting_primary' 2021-07-26T06:23:47.959 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:47.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2021-07-26T06:23:47.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.c"' 2021-07-26T06:23:47.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.c 2021-07-26T06:23:47.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.c scrub 2021-07-26T06:23:48.087 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:48.087 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:48.087 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:08.086646+0000" 2021-07-26T06:23:48.088 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:48.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:48.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:48.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.e") | .acting_primary' 2021-07-26T06:23:48.363 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:48.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2021-07-26T06:23:48.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.e"' 2021-07-26T06:23:48.383 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.e 2021-07-26T06:23:48.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.e scrub 2021-07-26T06:23:48.495 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:48.500 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:48.500 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:08.491174+0000" 2021-07-26T06:23:48.500 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:48.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:48.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:48.502 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.d") | .acting_primary' 2021-07-26T06:23:48.767 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:48.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2021-07-26T06:23:48.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.d"' 2021-07-26T06:23:48.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.d 2021-07-26T06:23:48.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.d scrub 2021-07-26T06:23:48.927 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:48.928 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:48.928 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:08.927807+0000" 2021-07-26T06:23:48.928 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:48.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:48.986 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:48.987 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.f") | .acting_primary' 2021-07-26T06:23:49.218 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:49.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=0 2021-07-26T06:23:49.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.f"' 2021-07-26T06:23:49.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.f 2021-07-26T06:23:49.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.f scrub 2021-07-26T06:23:49.335 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:49.335 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:49.336 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:09.332907+0000" 2021-07-26T06:23:49.336 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:49.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:49.357 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:49.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.d") | .acting_primary' 2021-07-26T06:23:49.616 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:49.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=2 2021-07-26T06:23:49.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.d"' 2021-07-26T06:23:49.617 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.d 2021-07-26T06:23:49.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.d scrub 2021-07-26T06:23:49.737 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:49.737 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:49.738 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:09.737723+0000" 2021-07-26T06:23:49.738 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:49.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:49.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:49.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.e") | .acting_primary' 2021-07-26T06:23:50.011 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:50.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=2 2021-07-26T06:23:50.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.e"' 2021-07-26T06:23:50.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.e 2021-07-26T06:23:50.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.e scrub 2021-07-26T06:23:50.144 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:50.145 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:50.168 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:10.142339+0000" 2021-07-26T06:23:50.168 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:50.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:50.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:50.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.e") | .acting_primary' 2021-07-26T06:23:50.452 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:50.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=2 2021-07-26T06:23:50.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.e"' 2021-07-26T06:23:50.453 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.e 2021-07-26T06:23:50.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.e scrub 2021-07-26T06:23:50.550 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:50.561 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:50.561 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:10.549906+0000" 2021-07-26T06:23:50.562 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:50.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:50.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:50.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.c") | .acting_primary' 2021-07-26T06:23:50.834 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:50.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2021-07-26T06:23:50.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.c"' 2021-07-26T06:23:50.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.c 2021-07-26T06:23:50.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.c scrub 2021-07-26T06:23:50.958 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:50.959 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:50.959 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:10.952612+0000" 2021-07-26T06:23:50.959 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:50.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:50.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:50.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.f") | .acting_primary' 2021-07-26T06:23:51.227 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:51.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=0 2021-07-26T06:23:51.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.f"' 2021-07-26T06:23:51.242 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.f 2021-07-26T06:23:51.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.f scrub 2021-07-26T06:23:51.406 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:51.406 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:51.418 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:11.400753+0000" 2021-07-26T06:23:51.418 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:51.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:51.424 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:51.424 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.9") | .acting_primary' 2021-07-26T06:23:51.675 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:51.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2021-07-26T06:23:51.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.9"' 2021-07-26T06:23:51.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.9 2021-07-26T06:23:51.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.9 scrub 2021-07-26T06:23:51.839 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:51.839 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:51.840 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:11.805964+0000" 2021-07-26T06:23:51.840 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:51.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:51.851 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:51.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.b") | .acting_primary' 2021-07-26T06:23:52.082 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:52.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2021-07-26T06:23:52.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.b"' 2021-07-26T06:23:52.128 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.b 2021-07-26T06:23:52.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.b scrub 2021-07-26T06:23:52.219 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:52.219 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:52.220 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:12.209906+0000" 2021-07-26T06:23:52.220 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:52.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:52.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:52.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.8") | .acting_primary' 2021-07-26T06:23:52.543 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:52.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=0 2021-07-26T06:23:52.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.8"' 2021-07-26T06:23:52.544 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.8 2021-07-26T06:23:52.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.8 scrub 2021-07-26T06:23:52.655 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:52.655 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:52.655 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:12.656716+0000" 2021-07-26T06:23:52.655 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:52.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:52.677 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:52.677 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.8") | .acting_primary' 2021-07-26T06:23:52.933 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:52.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2021-07-26T06:23:52.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.8"' 2021-07-26T06:23:52.955 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.8 2021-07-26T06:23:52.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.8 scrub 2021-07-26T06:23:53.074 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:53.099 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:53.099 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:13.063996+0000" 2021-07-26T06:23:53.114 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:53.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:53.116 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:53.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.a") | .acting_primary' 2021-07-26T06:23:53.388 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:53.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2021-07-26T06:23:53.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.a"' 2021-07-26T06:23:53.389 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.a 2021-07-26T06:23:53.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.a scrub 2021-07-26T06:23:53.525 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:53.526 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:53.526 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:13.466481+0000" 2021-07-26T06:23:53.562 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:53.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:53.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:53.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.9") | .acting_primary' 2021-07-26T06:23:53.743 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:53.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2021-07-26T06:23:53.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.9"' 2021-07-26T06:23:53.750 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.9 2021-07-26T06:23:53.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.9 scrub 2021-07-26T06:23:53.882 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:53.883 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:53.883 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:13.870023+0000" 2021-07-26T06:23:53.896 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:53.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:53.897 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:53.898 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.b") | .acting_primary' 2021-07-26T06:23:54.147 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:54.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2021-07-26T06:23:54.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.b"' 2021-07-26T06:23:54.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.b 2021-07-26T06:23:54.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.b scrub 2021-07-26T06:23:54.275 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:54.275 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:54.275 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:14.275766+0000" 2021-07-26T06:23:54.276 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:54.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:54.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:54.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.9") | .acting_primary' 2021-07-26T06:23:54.556 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:54.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2021-07-26T06:23:54.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.9"' 2021-07-26T06:23:54.562 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.9 2021-07-26T06:23:54.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.9 scrub 2021-07-26T06:23:54.682 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:54.683 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:54.683 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:14.682271+0000" 2021-07-26T06:23:54.683 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:54.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:54.704 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:54.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.a") | .acting_primary' 2021-07-26T06:23:54.955 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:54.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2021-07-26T06:23:54.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.a"' 2021-07-26T06:23:54.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.a 2021-07-26T06:23:54.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.a scrub 2021-07-26T06:23:55.093 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:55.093 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:55.094 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:15.086572+0000" 2021-07-26T06:23:55.094 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:55.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:55.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:55.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.a") | .acting_primary' 2021-07-26T06:23:55.372 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:55.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2021-07-26T06:23:55.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.a"' 2021-07-26T06:23:55.373 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.a 2021-07-26T06:23:55.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.a scrub 2021-07-26T06:23:55.491 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:55.492 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:55.492 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:15.493274+0000" 2021-07-26T06:23:55.492 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:55.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:55.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:55.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.8") | .acting_primary' 2021-07-26T06:23:55.771 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:55.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2021-07-26T06:23:55.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.8"' 2021-07-26T06:23:55.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.8 2021-07-26T06:23:55.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.8 scrub 2021-07-26T06:23:55.898 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:55.899 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:55.899 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:15.899737+0000" 2021-07-26T06:23:55.912 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:55.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:55.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:55.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.b") | .acting_primary' 2021-07-26T06:23:56.199 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:56.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2021-07-26T06:23:56.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.b"' 2021-07-26T06:23:56.200 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.b 2021-07-26T06:23:56.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.b scrub 2021-07-26T06:23:56.319 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:56.319 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:56.319 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:16.315795+0000" 2021-07-26T06:23:56.320 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:56.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:56.328 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:56.328 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.5") | .acting_primary' 2021-07-26T06:23:56.605 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:56.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2021-07-26T06:23:56.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.5"' 2021-07-26T06:23:56.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.5 2021-07-26T06:23:56.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.5 scrub 2021-07-26T06:23:56.719 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:56.720 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:56.720 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:16.721229+0000" 2021-07-26T06:23:56.720 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:56.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:56.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:56.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.7") | .acting_primary' 2021-07-26T06:23:56.996 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:57.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2021-07-26T06:23:57.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.7"' 2021-07-26T06:23:57.013 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.7 2021-07-26T06:23:57.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.7 scrub 2021-07-26T06:23:57.127 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:57.127 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:57.135 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:17.125434+0000" 2021-07-26T06:23:57.135 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:57.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:57.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:57.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.4") | .acting_primary' 2021-07-26T06:23:57.399 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:57.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2021-07-26T06:23:57.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.4"' 2021-07-26T06:23:57.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.4 2021-07-26T06:23:57.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.4 scrub 2021-07-26T06:23:57.541 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:57.541 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:57.542 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:17.542613+0000" 2021-07-26T06:23:57.542 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:57.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:57.552 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:57.552 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.4") | .acting_primary' 2021-07-26T06:23:57.819 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:57.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2021-07-26T06:23:57.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.4"' 2021-07-26T06:23:57.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.4 2021-07-26T06:23:57.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.4 scrub 2021-07-26T06:23:57.967 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:57.968 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:57.968 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:17.948360+0000" 2021-07-26T06:23:57.968 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:57.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:57.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:57.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.6") | .acting_primary' 2021-07-26T06:23:58.223 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:58.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2021-07-26T06:23:58.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.6"' 2021-07-26T06:23:58.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.6 2021-07-26T06:23:58.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.6 scrub 2021-07-26T06:23:58.400 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:58.400 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:58.401 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:18.378551+0000" 2021-07-26T06:23:58.401 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:58.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:58.403 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:58.403 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.5") | .acting_primary' 2021-07-26T06:23:58.654 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:58.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2021-07-26T06:23:58.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.5"' 2021-07-26T06:23:58.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.5 2021-07-26T06:23:58.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.5 scrub 2021-07-26T06:23:58.787 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:58.803 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:58.803 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:18.784378+0000" 2021-07-26T06:23:58.803 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:58.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:58.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:58.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.1") | .acting_primary' 2021-07-26T06:23:59.067 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:59.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2021-07-26T06:23:59.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.1"' 2021-07-26T06:23:59.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.1 2021-07-26T06:23:59.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.1 scrub 2021-07-26T06:23:59.223 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:59.223 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:59.224 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:19.192427+0000" 2021-07-26T06:23:59.224 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:59.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:59.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:59.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.2") | .acting_primary' 2021-07-26T06:23:59.469 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:59.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2021-07-26T06:23:59.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.2"' 2021-07-26T06:23:59.494 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.2 2021-07-26T06:23:59.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.2 scrub 2021-07-26T06:23:59.602 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:23:59.631 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:23:59.632 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:19.598314+0000" 2021-07-26T06:23:59.632 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:23:59.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:23:59.649 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:23:59.649 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.3") | .acting_primary' 2021-07-26T06:23:59.878 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:23:59.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2021-07-26T06:23:59.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.3"' 2021-07-26T06:23:59.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.3 2021-07-26T06:23:59.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.3 scrub 2021-07-26T06:24:00.002 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:00.002 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:00.011 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:20.004041+0000" 2021-07-26T06:24:00.012 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:00.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:00.013 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:00.013 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.0") | .acting_primary' 2021-07-26T06:24:00.277 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:00.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2021-07-26T06:24:00.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.0"' 2021-07-26T06:24:00.296 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.0 2021-07-26T06:24:00.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.0 scrub 2021-07-26T06:24:00.445 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:00.445 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:00.445 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:20.410121+0000" 2021-07-26T06:24:00.445 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:00.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:00.485 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:00.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.3") | .acting_primary' 2021-07-26T06:24:00.683 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:00.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2021-07-26T06:24:00.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.3"' 2021-07-26T06:24:00.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.3 2021-07-26T06:24:00.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.3 scrub 2021-07-26T06:24:00.816 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:00.817 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:00.817 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:20.813003+0000" 2021-07-26T06:24:00.817 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:00.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:00.845 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:00.845 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.2") | .acting_primary' 2021-07-26T06:24:01.087 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:01.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2021-07-26T06:24:01.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.2"' 2021-07-26T06:24:01.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.2 2021-07-26T06:24:01.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.2 scrub 2021-07-26T06:24:01.219 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:01.219 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:01.219 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:21.215259+0000" 2021-07-26T06:24:01.220 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:01.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:01.249 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:01.250 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.2") | .acting_primary' 2021-07-26T06:24:01.497 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:01.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=0 2021-07-26T06:24:01.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.2"' 2021-07-26T06:24:01.514 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.2 2021-07-26T06:24:01.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.2 scrub 2021-07-26T06:24:01.619 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:01.687 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:01.687 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:21.618489+0000" 2021-07-26T06:24:01.687 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:01.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:01.689 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:01.689 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.1") | .acting_primary' 2021-07-26T06:24:01.904 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:01.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=2 2021-07-26T06:24:01.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.1"' 2021-07-26T06:24:01.905 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.1 2021-07-26T06:24:01.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.1 scrub 2021-07-26T06:24:02.021 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:02.022 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:02.022 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:22.023080+0000" 2021-07-26T06:24:02.022 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:02.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:02.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:02.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.0") | .acting_primary' 2021-07-26T06:24:02.367 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:02.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2021-07-26T06:24:02.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.0"' 2021-07-26T06:24:02.368 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.0 2021-07-26T06:24:02.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.0 scrub 2021-07-26T06:24:02.429 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:02.429 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:02.429 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:22.429205+0000" 2021-07-26T06:24:02.429 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:02.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:02.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:02.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.3") | .acting_primary' 2021-07-26T06:24:02.704 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:02.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2021-07-26T06:24:02.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.3"' 2021-07-26T06:24:02.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.3 2021-07-26T06:24:02.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.3 scrub 2021-07-26T06:24:02.833 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:02.833 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:02.833 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:22.833352+0000" 2021-07-26T06:24:02.833 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:02.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:02.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:02.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.0") | .acting_primary' 2021-07-26T06:24:03.104 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:03.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2021-07-26T06:24:03.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.0"' 2021-07-26T06:24:03.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.0 2021-07-26T06:24:03.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.0 scrub 2021-07-26T06:24:03.235 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:03.236 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:03.236 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:23.235056+0000" 2021-07-26T06:24:03.263 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:03.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:03.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:03.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.1") | .acting_primary' 2021-07-26T06:24:03.533 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:03.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2021-07-26T06:24:03.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.1"' 2021-07-26T06:24:03.570 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.1 2021-07-26T06:24:03.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.1 scrub 2021-07-26T06:24:03.659 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:03.659 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:03.670 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:23.660812+0000" 2021-07-26T06:24:03.671 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:03.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:03.679 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:03.679 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.6") | .acting_primary' 2021-07-26T06:24:03.935 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:03.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=0 2021-07-26T06:24:03.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.6"' 2021-07-26T06:24:03.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.6 2021-07-26T06:24:03.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.6 scrub 2021-07-26T06:24:04.066 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:04.067 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:04.067 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:24.062295+0000" 2021-07-26T06:24:04.078 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:04.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:04.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:04.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.4") | .acting_primary' 2021-07-26T06:24:04.345 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:04.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2021-07-26T06:24:04.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.4"' 2021-07-26T06:24:04.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.4 2021-07-26T06:24:04.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.4 scrub 2021-07-26T06:24:04.467 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:04.467 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:04.480 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:24.466498+0000" 2021-07-26T06:24:04.481 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:04.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:04.482 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:04.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.7") | .acting_primary' 2021-07-26T06:24:04.742 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:04.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2021-07-26T06:24:04.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.7"' 2021-07-26T06:24:04.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.7 2021-07-26T06:24:04.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.7 scrub 2021-07-26T06:24:04.873 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:04.873 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:04.874 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:24.872310+0000" 2021-07-26T06:24:04.874 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:04.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:04.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:04.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.7") | .acting_primary' 2021-07-26T06:24:05.152 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:05.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2021-07-26T06:24:05.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.7"' 2021-07-26T06:24:05.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.7 2021-07-26T06:24:05.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.7 scrub 2021-07-26T06:24:05.275 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:05.275 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:05.281 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:25.276257+0000" 2021-07-26T06:24:05.281 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:05.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:05.332 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:05.332 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.5") | .acting_primary' 2021-07-26T06:24:05.567 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:05.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2021-07-26T06:24:05.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.5"' 2021-07-26T06:24:05.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.5 2021-07-26T06:24:05.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.5 scrub 2021-07-26T06:24:05.687 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:05.689 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:05.689 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:25.680858+0000" 2021-07-26T06:24:05.689 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:05.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2021-07-26T06:24:05.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2021-07-26T06:24:05.691 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.6") | .acting_primary' 2021-07-26T06:24:05.958 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:05.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2021-07-26T06:24:05.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.6"' 2021-07-26T06:24:05.985 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.6 2021-07-26T06:24:05.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.6 scrub 2021-07-26T06:24:06.083 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:06.083 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:24:06.084 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:22:26.083082+0000" 2021-07-26T06:24:06.085 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:06.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:91: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:24:06.355 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:24:06.367 INFO:tasks.workunit.client.0.smithi071.stdout:3.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:28.690937+0000 43'67 45:81 [3,1,0] 3 [3,1,0] 3 0'0 2021-07-19T06:22:06.851154+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.367 INFO:tasks.workunit.client.0.smithi071.stdout:1.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:24.487089+0000 43'62 45:84 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:07.255765+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.367 INFO:tasks.workunit.client.0.smithi071.stdout:2.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:26.399489+0000 43'60 45:78 [2,0,5] 2 [2,0,5] 2 0'0 2021-07-19T06:22:07.681429+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.368 INFO:tasks.workunit.client.0.smithi071.stdout:3.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:28.694238+0000 43'60 45:73 [5,3,1] 5 [5,3,1] 5 0'0 2021-07-19T06:22:08.086646+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.381 INFO:tasks.workunit.client.0.smithi071.stdout:1.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:24.160101+0000 43'69 45:90 [1,4,2] 1 [1,4,2] 1 0'0 2021-07-19T06:22:08.491174+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.381 INFO:tasks.workunit.client.0.smithi071.stdout:2.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:26.500677+0000 43'67 45:83 [5,1,3] 5 [5,1,3] 5 0'0 2021-07-19T06:22:08.927807+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.381 INFO:tasks.workunit.client.0.smithi071.stdout:3.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:28.693011+0000 43'62 45:76 [0,1,4] 0 [0,1,4] 0 0'0 2021-07-19T06:22:09.332907+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.382 INFO:tasks.workunit.client.0.smithi071.stdout:1.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:24.195729+0000 43'67 45:89 [2,4,1] 2 [2,4,1] 2 0'0 2021-07-19T06:22:09.737723+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.382 INFO:tasks.workunit.client.0.smithi071.stdout:2.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:26.500699+0000 43'69 45:87 [2,3,5] 2 [2,3,5] 2 0'0 2021-07-19T06:22:10.142339+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.394 INFO:tasks.workunit.client.0.smithi071.stdout:3.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:28.691787+0000 43'69 45:83 [2,4,1] 2 [2,4,1] 2 0'0 2021-07-19T06:22:10.549906+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.394 INFO:tasks.workunit.client.0.smithi071.stdout:1.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:24.621336+0000 43'60 45:82 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:10.952612+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.395 INFO:tasks.workunit.client.0.smithi071.stdout:2.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:26.542415+0000 43'62 45:80 [0,5,4] 0 [0,5,4] 0 0'0 2021-07-19T06:22:11.400753+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.395 INFO:tasks.workunit.client.0.smithi071.stdout:3.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:28.693145+0000 43'61 45:75 [4,2,5] 4 [4,2,5] 4 0'0 2021-07-19T06:22:11.805964+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.395 INFO:tasks.workunit.client.0.smithi071.stdout:1.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:23.647423+0000 43'65 45:86 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:12.209906+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.395 INFO:tasks.workunit.client.0.smithi071.stdout:2.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:26.399792+0000 43'66 45:84 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-19T06:22:12.656716+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.396 INFO:tasks.workunit.client.0.smithi071.stdout:3.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:28.695650+0000 43'66 45:80 [4,1,2] 4 [4,1,2] 4 0'0 2021-07-19T06:22:13.063996+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.396 INFO:tasks.workunit.client.0.smithi071.stdout:1.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:24.159370+0000 43'61 45:82 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:13.466481+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.396 INFO:tasks.workunit.client.0.smithi071.stdout:2.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:27.381838+0000 43'61 45:79 [4,1,0] 4 [4,1,0] 4 0'0 2021-07-19T06:22:13.870023+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.396 INFO:tasks.workunit.client.0.smithi071.stdout:3.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:28.691693+0000 43'65 45:79 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:14.275766+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.397 INFO:tasks.workunit.client.0.smithi071.stdout:1.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:24.159919+0000 43'61 45:82 [1,4,5] 1 [1,4,5] 1 0'0 2021-07-19T06:22:14.682271+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.397 INFO:tasks.workunit.client.0.smithi071.stdout:2.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:26.324693+0000 43'61 45:78 [1,5,0] 1 [1,5,0] 1 0'0 2021-07-19T06:22:15.086572+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.397 INFO:tasks.workunit.client.0.smithi071.stdout:3.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:28.691424+0000 43'61 45:75 [3,1,5] 3 [3,1,5] 3 0'0 2021-07-19T06:22:15.493274+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.423 INFO:tasks.workunit.client.0.smithi071.stdout:1.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:24.346599+0000 43'66 45:86 [5,4,3] 5 [5,4,3] 5 0'0 2021-07-19T06:22:15.899737+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.423 INFO:tasks.workunit.client.0.smithi071.stdout:2.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:27.349429+0000 43'65 45:83 [3,2,5] 3 [3,2,5] 3 0'0 2021-07-19T06:22:16.315795+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.423 INFO:tasks.workunit.client.0.smithi071.stdout:3.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:28.692572+0000 43'73 45:86 [5,3,4] 5 [5,3,4] 5 0'0 2021-07-19T06:22:16.721229+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.424 INFO:tasks.workunit.client.0.smithi071.stdout:1.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:23.954687+0000 43'53 45:74 [1,3,0] 1 [1,3,0] 1 0'0 2021-07-19T06:22:17.125434+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.424 INFO:tasks.workunit.client.0.smithi071.stdout:2.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:26.399721+0000 43'51 45:68 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:17.542613+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.424 INFO:tasks.workunit.client.0.smithi071.stdout:3.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:28.693313+0000 43'51 45:65 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:17.948360+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.424 INFO:tasks.workunit.client.0.smithi071.stdout:1.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:24.234706+0000 43'64 45:86 [3,4,2] 3 [3,4,2] 3 0'0 2021-07-19T06:22:18.378551+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.425 INFO:tasks.workunit.client.0.smithi071.stdout:2.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:27.381761+0000 43'73 45:91 [3,4,0] 3 [3,4,0] 3 0'0 2021-07-19T06:22:18.784378+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.425 INFO:tasks.workunit.client.0.smithi071.stdout:1.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:24.346071+0000 43'67 45:87 [5,0,4] 5 [5,0,4] 5 0'0 2021-07-19T06:22:19.192427+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.425 INFO:tasks.workunit.client.0.smithi071.stdout:2.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:26.357673+0000 43'55 45:71 [5,1,0] 5 [5,1,0] 5 0'0 2021-07-19T06:22:19.598314+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.425 INFO:tasks.workunit.client.0.smithi071.stdout:3.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:28.691968+0000 43'54 45:68 [4,0,5] 4 [4,0,5] 4 0'0 2021-07-19T06:22:20.004041+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.425 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:23.647564+0000 43'72 45:93 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:20.410121+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.426 INFO:tasks.workunit.client.0.smithi071.stdout:2.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:26.542345+0000 43'54 45:70 [5,4,1] 5 [5,4,1] 5 0'0 2021-07-19T06:22:20.813003+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.426 INFO:tasks.workunit.client.0.smithi071.stdout:3.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:28.690784+0000 43'55 45:69 [3,5,0] 3 [3,5,0] 3 0'0 2021-07-19T06:22:21.215259+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.426 INFO:tasks.workunit.client.0.smithi071.stdout:1.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:24.346334+0000 43'55 44:76 [0,4,1] 0 [0,4,1] 0 0'0 2021-07-26T06:16:22.467639+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.456 INFO:tasks.workunit.client.0.smithi071.stdout:2.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:26.399501+0000 43'67 45:85 [2,5,0] 2 [2,5,0] 2 0'0 2021-07-19T06:22:22.023080+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.456 INFO:tasks.workunit.client.0.smithi071.stdout:3.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:28.692254+0000 43'72 44:85 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-26T06:16:26.500119+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.457 INFO:tasks.workunit.client.0.smithi071.stdout:1.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:24.234654+0000 43'54 44:74 [4,1,2] 4 [4,1,2] 4 0'0 2021-07-26T06:16:22.467639+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.457 INFO:tasks.workunit.client.0.smithi071.stdout:2.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:27.213682+0000 43'72 45:90 [3,5,1] 3 [3,5,1] 3 0'0 2021-07-19T06:22:23.235056+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.457 INFO:tasks.workunit.client.0.smithi071.stdout:3.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:28.692028+0000 43'67 44:80 [4,3,5] 4 [4,3,5] 4 0'0 2021-07-26T06:16:26.500119+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.457 INFO:tasks.workunit.client.0.smithi071.stdout:3.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:28.693466+0000 43'64 44:77 [0,1,4] 0 [0,1,4] 0 0'0 2021-07-26T06:16:26.500119+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.458 INFO:tasks.workunit.client.0.smithi071.stdout:1.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:24.487000+0000 43'51 44:72 [3,0,1] 3 [3,0,1] 3 0'0 2021-07-26T06:16:22.467639+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.458 INFO:tasks.workunit.client.0.smithi071.stdout:2.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:27.381839+0000 43'53 44:70 [3,0,2] 3 [3,0,2] 3 0'0 2021-07-26T06:16:24.376841+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.458 INFO:tasks.workunit.client.0.smithi071.stdout:3.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:28.691181+0000 43'53 44:66 [4,3,0] 4 [4,3,0] 4 0'0 2021-07-26T06:16:26.500119+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:06.458 INFO:tasks.workunit.client.0.smithi071.stdout:1.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:24.854235+0000 43'73 44:93 [4,2,5] 4 [4,2,5] 4 0'0 2021-07-26T06:16:22.467639+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:06.458 INFO:tasks.workunit.client.0.smithi071.stdout:2.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:26.542414+0000 43'64 44:80 [1,0,4] 1 [1,0,4] 1 0'0 2021-07-26T06:16:24.376841+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:06.459 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:24:06.459 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:24:06.461 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:06.461 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:93: TEST_recover_unexpected: jq .osd_max_scrubs 2021-07-26T06:24:06.461 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:93: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:06.461 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:06.462 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:06.462 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:06.462 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:06.462 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:06.463 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:06.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:93: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:06.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:93: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:06.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:93: TEST_recover_unexpected: max=4 2021-07-26T06:24:06.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:94: TEST_recover_unexpected: '[' 4 '!=' '4]' 2021-07-26T06:24:06.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh: line 94: [: missing `]' 2021-07-26T06:24:06.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:100: TEST_recover_unexpected: ceph osd unset noscrub 2021-07-26T06:24:06.871 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is unset 2021-07-26T06:24:06.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:102: TEST_recover_unexpected: ok=false 2021-07-26T06:24:06.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: seq 0 300 2021-07-26T06:24:06.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: for i in $(seq 0 300) 2021-07-26T06:24:06.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:105: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:24:07.237 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:24:07.237 INFO:tasks.workunit.client.0.smithi071.stdout:3.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:28.690937+0000 43'67 45:81 [3,1,0] 3 [3,1,0] 3 0'0 2021-07-19T06:22:06.851154+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.237 INFO:tasks.workunit.client.0.smithi071.stdout:1.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:24.487089+0000 43'62 45:84 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:07.255765+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.262 INFO:tasks.workunit.client.0.smithi071.stdout:2.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:26.399489+0000 43'60 45:78 [2,0,5] 2 [2,0,5] 2 0'0 2021-07-19T06:22:07.681429+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.263 INFO:tasks.workunit.client.0.smithi071.stdout:3.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:28.694238+0000 43'60 45:73 [5,3,1] 5 [5,3,1] 5 0'0 2021-07-19T06:22:08.086646+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.263 INFO:tasks.workunit.client.0.smithi071.stdout:1.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:24.160101+0000 43'69 45:90 [1,4,2] 1 [1,4,2] 1 0'0 2021-07-19T06:22:08.491174+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.263 INFO:tasks.workunit.client.0.smithi071.stdout:2.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:26.500677+0000 43'67 45:83 [5,1,3] 5 [5,1,3] 5 0'0 2021-07-19T06:22:08.927807+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.263 INFO:tasks.workunit.client.0.smithi071.stdout:3.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:28.693011+0000 43'62 45:76 [0,1,4] 0 [0,1,4] 0 0'0 2021-07-19T06:22:09.332907+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.264 INFO:tasks.workunit.client.0.smithi071.stdout:1.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:24.195729+0000 43'67 45:89 [2,4,1] 2 [2,4,1] 2 0'0 2021-07-19T06:22:09.737723+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.264 INFO:tasks.workunit.client.0.smithi071.stdout:2.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:26.500699+0000 43'69 45:87 [2,3,5] 2 [2,3,5] 2 0'0 2021-07-19T06:22:10.142339+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.264 INFO:tasks.workunit.client.0.smithi071.stdout:3.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:28.691787+0000 43'69 45:83 [2,4,1] 2 [2,4,1] 2 0'0 2021-07-19T06:22:10.549906+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.264 INFO:tasks.workunit.client.0.smithi071.stdout:1.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:24.621336+0000 43'60 45:82 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:10.952612+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.265 INFO:tasks.workunit.client.0.smithi071.stdout:2.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:26.542415+0000 43'62 45:80 [0,5,4] 0 [0,5,4] 0 0'0 2021-07-19T06:22:11.400753+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.265 INFO:tasks.workunit.client.0.smithi071.stdout:3.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:28.693145+0000 43'61 45:75 [4,2,5] 4 [4,2,5] 4 0'0 2021-07-19T06:22:11.805964+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.265 INFO:tasks.workunit.client.0.smithi071.stdout:1.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:23.647423+0000 43'65 45:86 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:12.209906+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.265 INFO:tasks.workunit.client.0.smithi071.stdout:2.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:26.399792+0000 43'66 45:84 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-19T06:22:12.656716+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.265 INFO:tasks.workunit.client.0.smithi071.stdout:3.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:28.695650+0000 43'66 45:80 [4,1,2] 4 [4,1,2] 4 0'0 2021-07-19T06:22:13.063996+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.266 INFO:tasks.workunit.client.0.smithi071.stdout:1.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:24.159370+0000 43'61 45:82 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:13.466481+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.266 INFO:tasks.workunit.client.0.smithi071.stdout:2.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:27.381838+0000 43'61 45:79 [4,1,0] 4 [4,1,0] 4 0'0 2021-07-19T06:22:13.870023+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.266 INFO:tasks.workunit.client.0.smithi071.stdout:3.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:28.691693+0000 43'65 45:79 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:14.275766+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.266 INFO:tasks.workunit.client.0.smithi071.stdout:1.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:24.159919+0000 43'61 45:82 [1,4,5] 1 [1,4,5] 1 0'0 2021-07-19T06:22:14.682271+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.267 INFO:tasks.workunit.client.0.smithi071.stdout:2.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:26.324693+0000 43'61 45:78 [1,5,0] 1 [1,5,0] 1 0'0 2021-07-19T06:22:15.086572+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.267 INFO:tasks.workunit.client.0.smithi071.stdout:3.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:28.691424+0000 43'61 45:75 [3,1,5] 3 [3,1,5] 3 0'0 2021-07-19T06:22:15.493274+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.267 INFO:tasks.workunit.client.0.smithi071.stdout:1.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:24.346599+0000 43'66 45:86 [5,4,3] 5 [5,4,3] 5 0'0 2021-07-19T06:22:15.899737+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.267 INFO:tasks.workunit.client.0.smithi071.stdout:2.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:27.349429+0000 43'65 45:83 [3,2,5] 3 [3,2,5] 3 0'0 2021-07-19T06:22:16.315795+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.268 INFO:tasks.workunit.client.0.smithi071.stdout:3.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:28.692572+0000 43'73 45:86 [5,3,4] 5 [5,3,4] 5 0'0 2021-07-19T06:22:16.721229+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.268 INFO:tasks.workunit.client.0.smithi071.stdout:1.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:23.954687+0000 43'53 45:74 [1,3,0] 1 [1,3,0] 1 0'0 2021-07-19T06:22:17.125434+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.277 INFO:tasks.workunit.client.0.smithi071.stdout:2.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:26.399721+0000 43'51 45:68 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:17.542613+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.277 INFO:tasks.workunit.client.0.smithi071.stdout:3.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:28.693313+0000 43'51 45:65 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:17.948360+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.278 INFO:tasks.workunit.client.0.smithi071.stdout:1.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:24.234706+0000 43'64 45:86 [3,4,2] 3 [3,4,2] 3 0'0 2021-07-19T06:22:18.378551+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.278 INFO:tasks.workunit.client.0.smithi071.stdout:2.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:27.381761+0000 43'73 45:91 [3,4,0] 3 [3,4,0] 3 0'0 2021-07-19T06:22:18.784378+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.344 INFO:tasks.workunit.client.0.smithi071.stdout:1.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:24.346071+0000 43'67 45:87 [5,0,4] 5 [5,0,4] 5 0'0 2021-07-19T06:22:19.192427+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.345 INFO:tasks.workunit.client.0.smithi071.stdout:2.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:26.357673+0000 43'55 45:71 [5,1,0] 5 [5,1,0] 5 0'0 2021-07-19T06:22:19.598314+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.345 INFO:tasks.workunit.client.0.smithi071.stdout:3.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:28.691968+0000 43'54 45:68 [4,0,5] 4 [4,0,5] 4 0'0 2021-07-19T06:22:20.004041+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.345 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:23.647564+0000 43'72 45:93 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:20.410121+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.345 INFO:tasks.workunit.client.0.smithi071.stdout:2.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:26.542345+0000 43'54 45:70 [5,4,1] 5 [5,4,1] 5 0'0 2021-07-19T06:22:20.813003+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.346 INFO:tasks.workunit.client.0.smithi071.stdout:3.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:28.690784+0000 43'55 45:69 [3,5,0] 3 [3,5,0] 3 0'0 2021-07-19T06:22:21.215259+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.346 INFO:tasks.workunit.client.0.smithi071.stdout:1.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:24.346334+0000 43'55 45:77 [0,4,1] 0 [0,4,1] 0 0'0 2021-07-19T06:22:21.618489+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.346 INFO:tasks.workunit.client.0.smithi071.stdout:2.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:26.399501+0000 43'67 45:85 [2,5,0] 2 [2,5,0] 2 0'0 2021-07-19T06:22:22.023080+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.380 INFO:tasks.workunit.client.0.smithi071.stdout:3.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:28.692254+0000 43'72 45:86 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:22.429205+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.380 INFO:tasks.workunit.client.0.smithi071.stdout:1.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:24.234654+0000 43'54 45:75 [4,1,2] 4 [4,1,2] 4 0'0 2021-07-19T06:22:22.833352+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.380 INFO:tasks.workunit.client.0.smithi071.stdout:2.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:27.213682+0000 43'72 45:90 [3,5,1] 3 [3,5,1] 3 0'0 2021-07-19T06:22:23.235056+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.380 INFO:tasks.workunit.client.0.smithi071.stdout:3.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:28.692028+0000 43'67 45:81 [4,3,5] 4 [4,3,5] 4 0'0 2021-07-19T06:22:23.660812+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.400 INFO:tasks.workunit.client.0.smithi071.stdout:3.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:28.693466+0000 43'64 45:78 [0,1,4] 0 [0,1,4] 0 0'0 2021-07-19T06:22:24.062295+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.400 INFO:tasks.workunit.client.0.smithi071.stdout:1.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:24.487000+0000 43'51 44:72 [3,0,1] 3 [3,0,1] 3 0'0 2021-07-26T06:16:22.467639+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.401 INFO:tasks.workunit.client.0.smithi071.stdout:2.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:27.381839+0000 43'53 44:70 [3,0,2] 3 [3,0,2] 3 0'0 2021-07-26T06:16:24.376841+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.401 INFO:tasks.workunit.client.0.smithi071.stdout:3.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:28.691181+0000 43'53 45:67 [4,3,0] 4 [4,3,0] 4 0'0 2021-07-19T06:22:25.276257+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:07.401 INFO:tasks.workunit.client.0.smithi071.stdout:1.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:24.854235+0000 43'73 44:93 [4,2,5] 4 [4,2,5] 4 0'0 2021-07-26T06:16:22.467639+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:07.463 INFO:tasks.workunit.client.0.smithi071.stdout:2.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:26.542414+0000 43'64 45:81 [1,0,4] 1 [1,0,4] 1 0'0 2021-07-19T06:22:26.083082+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:07.463 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:24:07.464 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:24:07.477 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:07.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:24:07.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: grep scrubbing 2021-07-26T06:24:07.510 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:07.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:110: TEST_recover_unexpected: sleep 1 2021-07-26T06:24:08.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: for i in $(seq 0 300) 2021-07-26T06:24:08.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:105: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:24:08.790 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:24:08.791 INFO:tasks.workunit.client.0.smithi071.stdout:3.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:28.690937+0000 43'67 45:81 [3,1,0] 3 [3,1,0] 3 0'0 2021-07-19T06:22:06.851154+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.791 INFO:tasks.workunit.client.0.smithi071.stdout:1.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:24.487089+0000 43'62 45:84 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:07.255765+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.791 INFO:tasks.workunit.client.0.smithi071.stdout:2.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:26.399489+0000 43'60 45:78 [2,0,5] 2 [2,0,5] 2 0'0 2021-07-19T06:22:07.681429+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.791 INFO:tasks.workunit.client.0.smithi071.stdout:3.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:28.694238+0000 43'60 45:73 [5,3,1] 5 [5,3,1] 5 0'0 2021-07-19T06:22:08.086646+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.792 INFO:tasks.workunit.client.0.smithi071.stdout:1.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:24.160101+0000 43'69 45:90 [1,4,2] 1 [1,4,2] 1 0'0 2021-07-19T06:22:08.491174+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.792 INFO:tasks.workunit.client.0.smithi071.stdout:2.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:26.500677+0000 43'67 45:83 [5,1,3] 5 [5,1,3] 5 0'0 2021-07-19T06:22:08.927807+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.792 INFO:tasks.workunit.client.0.smithi071.stdout:3.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:28.693011+0000 43'62 45:76 [0,1,4] 0 [0,1,4] 0 0'0 2021-07-19T06:22:09.332907+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.792 INFO:tasks.workunit.client.0.smithi071.stdout:1.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:24.195729+0000 43'67 45:89 [2,4,1] 2 [2,4,1] 2 0'0 2021-07-19T06:22:09.737723+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.793 INFO:tasks.workunit.client.0.smithi071.stdout:2.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:26.500699+0000 43'69 45:87 [2,3,5] 2 [2,3,5] 2 0'0 2021-07-19T06:22:10.142339+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.793 INFO:tasks.workunit.client.0.smithi071.stdout:3.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:28.691787+0000 43'69 45:83 [2,4,1] 2 [2,4,1] 2 0'0 2021-07-19T06:22:10.549906+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.793 INFO:tasks.workunit.client.0.smithi071.stdout:1.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:24.621336+0000 43'60 45:82 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:10.952612+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.793 INFO:tasks.workunit.client.0.smithi071.stdout:2.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:26.542415+0000 43'62 45:80 [0,5,4] 0 [0,5,4] 0 0'0 2021-07-19T06:22:11.400753+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.794 INFO:tasks.workunit.client.0.smithi071.stdout:3.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:28.693145+0000 43'61 45:75 [4,2,5] 4 [4,2,5] 4 0'0 2021-07-19T06:22:11.805964+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.806 INFO:tasks.workunit.client.0.smithi071.stdout:1.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:23.647423+0000 43'65 45:86 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:12.209906+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.806 INFO:tasks.workunit.client.0.smithi071.stdout:2.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:26.399792+0000 43'66 45:84 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-19T06:22:12.656716+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.806 INFO:tasks.workunit.client.0.smithi071.stdout:3.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:28.695650+0000 43'66 45:80 [4,1,2] 4 [4,1,2] 4 0'0 2021-07-19T06:22:13.063996+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.807 INFO:tasks.workunit.client.0.smithi071.stdout:1.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:24.159370+0000 43'61 45:82 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:13.466481+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.807 INFO:tasks.workunit.client.0.smithi071.stdout:2.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:27.381838+0000 43'61 45:79 [4,1,0] 4 [4,1,0] 4 0'0 2021-07-19T06:22:13.870023+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.807 INFO:tasks.workunit.client.0.smithi071.stdout:3.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:28.691693+0000 43'65 45:79 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:14.275766+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.807 INFO:tasks.workunit.client.0.smithi071.stdout:1.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:24.159919+0000 43'61 45:82 [1,4,5] 1 [1,4,5] 1 0'0 2021-07-19T06:22:14.682271+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.808 INFO:tasks.workunit.client.0.smithi071.stdout:2.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:26.324693+0000 43'61 45:78 [1,5,0] 1 [1,5,0] 1 0'0 2021-07-19T06:22:15.086572+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.808 INFO:tasks.workunit.client.0.smithi071.stdout:3.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:28.691424+0000 43'61 45:75 [3,1,5] 3 [3,1,5] 3 0'0 2021-07-19T06:22:15.493274+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.808 INFO:tasks.workunit.client.0.smithi071.stdout:1.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:24.346599+0000 43'66 45:86 [5,4,3] 5 [5,4,3] 5 0'0 2021-07-19T06:22:15.899737+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.808 INFO:tasks.workunit.client.0.smithi071.stdout:2.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:27.349429+0000 43'65 45:83 [3,2,5] 3 [3,2,5] 3 0'0 2021-07-19T06:22:16.315795+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.809 INFO:tasks.workunit.client.0.smithi071.stdout:3.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:28.692572+0000 43'73 45:86 [5,3,4] 5 [5,3,4] 5 0'0 2021-07-19T06:22:16.721229+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.809 INFO:tasks.workunit.client.0.smithi071.stdout:1.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:23.954687+0000 43'53 45:74 [1,3,0] 1 [1,3,0] 1 0'0 2021-07-19T06:22:17.125434+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.809 INFO:tasks.workunit.client.0.smithi071.stdout:2.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:26.399721+0000 43'51 45:68 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:17.542613+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.809 INFO:tasks.workunit.client.0.smithi071.stdout:3.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:28.693313+0000 43'51 45:65 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:17.948360+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.810 INFO:tasks.workunit.client.0.smithi071.stdout:1.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:24.234706+0000 43'64 45:86 [3,4,2] 3 [3,4,2] 3 0'0 2021-07-19T06:22:18.378551+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.810 INFO:tasks.workunit.client.0.smithi071.stdout:2.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:27.381761+0000 43'73 45:91 [3,4,0] 3 [3,4,0] 3 0'0 2021-07-19T06:22:18.784378+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.810 INFO:tasks.workunit.client.0.smithi071.stdout:1.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:24.346071+0000 43'67 45:87 [5,0,4] 5 [5,0,4] 5 0'0 2021-07-19T06:22:19.192427+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.810 INFO:tasks.workunit.client.0.smithi071.stdout:2.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:26.357673+0000 43'55 45:71 [5,1,0] 5 [5,1,0] 5 0'0 2021-07-19T06:22:19.598314+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.811 INFO:tasks.workunit.client.0.smithi071.stdout:3.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:28.691968+0000 43'54 45:68 [4,0,5] 4 [4,0,5] 4 0'0 2021-07-19T06:22:20.004041+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.811 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:23.647564+0000 43'72 45:93 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:20.410121+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.811 INFO:tasks.workunit.client.0.smithi071.stdout:2.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:26.542345+0000 43'54 45:70 [5,4,1] 5 [5,4,1] 5 0'0 2021-07-19T06:22:20.813003+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.811 INFO:tasks.workunit.client.0.smithi071.stdout:3.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:28.690784+0000 43'55 45:69 [3,5,0] 3 [3,5,0] 3 0'0 2021-07-19T06:22:21.215259+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.812 INFO:tasks.workunit.client.0.smithi071.stdout:1.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:24.346334+0000 43'55 45:77 [0,4,1] 0 [0,4,1] 0 0'0 2021-07-19T06:22:21.618489+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.812 INFO:tasks.workunit.client.0.smithi071.stdout:2.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:26.399501+0000 43'67 45:85 [2,5,0] 2 [2,5,0] 2 0'0 2021-07-19T06:22:22.023080+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.812 INFO:tasks.workunit.client.0.smithi071.stdout:3.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:28.692254+0000 43'72 45:86 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:22.429205+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.813 INFO:tasks.workunit.client.0.smithi071.stdout:1.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:24.234654+0000 43'54 45:75 [4,1,2] 4 [4,1,2] 4 0'0 2021-07-19T06:22:22.833352+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.813 INFO:tasks.workunit.client.0.smithi071.stdout:2.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:27.213682+0000 43'72 45:90 [3,5,1] 3 [3,5,1] 3 0'0 2021-07-19T06:22:23.235056+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.813 INFO:tasks.workunit.client.0.smithi071.stdout:3.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:28.692028+0000 43'67 45:81 [4,3,5] 4 [4,3,5] 4 0'0 2021-07-19T06:22:23.660812+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.813 INFO:tasks.workunit.client.0.smithi071.stdout:3.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:28.693466+0000 43'64 45:78 [0,1,4] 0 [0,1,4] 0 0'0 2021-07-19T06:22:24.062295+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.814 INFO:tasks.workunit.client.0.smithi071.stdout:1.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:24.487000+0000 43'51 44:72 [3,0,1] 3 [3,0,1] 3 0'0 2021-07-26T06:16:22.467639+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.814 INFO:tasks.workunit.client.0.smithi071.stdout:2.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:27.381839+0000 43'53 44:70 [3,0,2] 3 [3,0,2] 3 0'0 2021-07-26T06:16:24.376841+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.814 INFO:tasks.workunit.client.0.smithi071.stdout:3.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:28.691181+0000 43'53 45:67 [4,3,0] 4 [4,3,0] 4 0'0 2021-07-19T06:22:25.276257+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:08.814 INFO:tasks.workunit.client.0.smithi071.stdout:1.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:24.854235+0000 43'73 44:93 [4,2,5] 4 [4,2,5] 4 0'0 2021-07-26T06:16:22.467639+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:08.815 INFO:tasks.workunit.client.0.smithi071.stdout:2.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:26.542414+0000 43'64 45:81 [1,0,4] 1 [1,0,4] 1 0'0 2021-07-19T06:22:26.083082+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:08.815 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:24:08.815 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:24:08.816 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:08.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:24:08.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: grep scrubbing 2021-07-26T06:24:09.084 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:09.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:110: TEST_recover_unexpected: sleep 1 2021-07-26T06:24:10.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: for i in $(seq 0 300) 2021-07-26T06:24:10.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:105: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:24:10.349 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:24:10.360 INFO:tasks.workunit.client.0.smithi071.stdout:3.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:28.690937+0000 43'67 45:81 [3,1,0] 3 [3,1,0] 3 0'0 2021-07-19T06:22:06.851154+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.361 INFO:tasks.workunit.client.0.smithi071.stdout:1.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:24.487089+0000 43'62 45:84 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:07.255765+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.361 INFO:tasks.workunit.client.0.smithi071.stdout:2.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:26.399489+0000 43'60 45:78 [2,0,5] 2 [2,0,5] 2 0'0 2021-07-19T06:22:07.681429+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.361 INFO:tasks.workunit.client.0.smithi071.stdout:3.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:28.694238+0000 43'60 45:73 [5,3,1] 5 [5,3,1] 5 0'0 2021-07-19T06:22:08.086646+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.362 INFO:tasks.workunit.client.0.smithi071.stdout:1.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:24.160101+0000 43'69 45:90 [1,4,2] 1 [1,4,2] 1 0'0 2021-07-19T06:22:08.491174+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.362 INFO:tasks.workunit.client.0.smithi071.stdout:2.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:26.500677+0000 43'67 45:83 [5,1,3] 5 [5,1,3] 5 0'0 2021-07-19T06:22:08.927807+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.362 INFO:tasks.workunit.client.0.smithi071.stdout:3.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:28.693011+0000 43'62 45:76 [0,1,4] 0 [0,1,4] 0 0'0 2021-07-19T06:22:09.332907+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.399 INFO:tasks.workunit.client.0.smithi071.stdout:1.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:24.195729+0000 43'67 45:89 [2,4,1] 2 [2,4,1] 2 0'0 2021-07-19T06:22:09.737723+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.399 INFO:tasks.workunit.client.0.smithi071.stdout:2.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:26.500699+0000 43'69 45:87 [2,3,5] 2 [2,3,5] 2 0'0 2021-07-19T06:22:10.142339+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.400 INFO:tasks.workunit.client.0.smithi071.stdout:3.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:28.691787+0000 43'69 45:83 [2,4,1] 2 [2,4,1] 2 0'0 2021-07-19T06:22:10.549906+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.400 INFO:tasks.workunit.client.0.smithi071.stdout:1.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:24.621336+0000 43'60 45:82 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:10.952612+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.400 INFO:tasks.workunit.client.0.smithi071.stdout:2.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:26.542415+0000 43'62 45:80 [0,5,4] 0 [0,5,4] 0 0'0 2021-07-19T06:22:11.400753+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.401 INFO:tasks.workunit.client.0.smithi071.stdout:3.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:28.693145+0000 43'61 45:75 [4,2,5] 4 [4,2,5] 4 0'0 2021-07-19T06:22:11.805964+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.401 INFO:tasks.workunit.client.0.smithi071.stdout:1.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:23.647423+0000 43'65 45:86 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:12.209906+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.412 INFO:tasks.workunit.client.0.smithi071.stdout:2.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:26.399792+0000 43'66 45:84 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-19T06:22:12.656716+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.419 INFO:tasks.workunit.client.0.smithi071.stdout:3.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:28.695650+0000 43'66 45:80 [4,1,2] 4 [4,1,2] 4 0'0 2021-07-19T06:22:13.063996+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.419 INFO:tasks.workunit.client.0.smithi071.stdout:1.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:24.159370+0000 43'61 45:82 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:13.466481+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.419 INFO:tasks.workunit.client.0.smithi071.stdout:2.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:27.381838+0000 43'61 45:79 [4,1,0] 4 [4,1,0] 4 0'0 2021-07-19T06:22:13.870023+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.420 INFO:tasks.workunit.client.0.smithi071.stdout:3.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:28.691693+0000 43'65 45:79 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:14.275766+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.420 INFO:tasks.workunit.client.0.smithi071.stdout:1.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:24.159919+0000 43'61 45:82 [1,4,5] 1 [1,4,5] 1 0'0 2021-07-19T06:22:14.682271+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.420 INFO:tasks.workunit.client.0.smithi071.stdout:2.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:26.324693+0000 43'61 45:78 [1,5,0] 1 [1,5,0] 1 0'0 2021-07-19T06:22:15.086572+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.420 INFO:tasks.workunit.client.0.smithi071.stdout:3.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:28.691424+0000 43'61 45:75 [3,1,5] 3 [3,1,5] 3 0'0 2021-07-19T06:22:15.493274+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.421 INFO:tasks.workunit.client.0.smithi071.stdout:1.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:24.346599+0000 43'66 45:86 [5,4,3] 5 [5,4,3] 5 0'0 2021-07-19T06:22:15.899737+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.421 INFO:tasks.workunit.client.0.smithi071.stdout:2.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:27.349429+0000 43'65 45:83 [3,2,5] 3 [3,2,5] 3 0'0 2021-07-19T06:22:16.315795+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.429 INFO:tasks.workunit.client.0.smithi071.stdout:3.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:28.692572+0000 43'73 45:86 [5,3,4] 5 [5,3,4] 5 0'0 2021-07-19T06:22:16.721229+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.429 INFO:tasks.workunit.client.0.smithi071.stdout:1.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:23.954687+0000 43'53 45:74 [1,3,0] 1 [1,3,0] 1 0'0 2021-07-19T06:22:17.125434+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.430 INFO:tasks.workunit.client.0.smithi071.stdout:2.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:26.399721+0000 43'51 45:68 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:17.542613+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.430 INFO:tasks.workunit.client.0.smithi071.stdout:3.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:28.693313+0000 43'51 45:65 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:17.948360+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.430 INFO:tasks.workunit.client.0.smithi071.stdout:1.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:24.234706+0000 43'64 45:86 [3,4,2] 3 [3,4,2] 3 0'0 2021-07-19T06:22:18.378551+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.432 INFO:tasks.workunit.client.0.smithi071.stdout:2.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:27.381761+0000 43'73 45:91 [3,4,0] 3 [3,4,0] 3 0'0 2021-07-19T06:22:18.784378+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.432 INFO:tasks.workunit.client.0.smithi071.stdout:1.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:24.346071+0000 43'67 45:87 [5,0,4] 5 [5,0,4] 5 0'0 2021-07-19T06:22:19.192427+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.433 INFO:tasks.workunit.client.0.smithi071.stdout:2.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:26.357673+0000 43'55 45:71 [5,1,0] 5 [5,1,0] 5 0'0 2021-07-19T06:22:19.598314+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.433 INFO:tasks.workunit.client.0.smithi071.stdout:3.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:28.691968+0000 43'54 45:68 [4,0,5] 4 [4,0,5] 4 0'0 2021-07-19T06:22:20.004041+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.433 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:23.647564+0000 43'72 45:93 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:20.410121+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.433 INFO:tasks.workunit.client.0.smithi071.stdout:2.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:26.542345+0000 43'54 45:70 [5,4,1] 5 [5,4,1] 5 0'0 2021-07-19T06:22:20.813003+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.434 INFO:tasks.workunit.client.0.smithi071.stdout:3.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:28.690784+0000 43'55 45:69 [3,5,0] 3 [3,5,0] 3 0'0 2021-07-19T06:22:21.215259+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.434 INFO:tasks.workunit.client.0.smithi071.stdout:1.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:24.346334+0000 43'55 45:77 [0,4,1] 0 [0,4,1] 0 0'0 2021-07-19T06:22:21.618489+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.434 INFO:tasks.workunit.client.0.smithi071.stdout:2.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:26.399501+0000 43'67 45:85 [2,5,0] 2 [2,5,0] 2 0'0 2021-07-19T06:22:22.023080+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.443 INFO:tasks.workunit.client.0.smithi071.stdout:3.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:28.692254+0000 43'72 45:86 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:22.429205+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.474 INFO:tasks.workunit.client.0.smithi071.stdout:1.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:24.234654+0000 43'54 45:75 [4,1,2] 4 [4,1,2] 4 0'0 2021-07-19T06:22:22.833352+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.474 INFO:tasks.workunit.client.0.smithi071.stdout:2.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:27.213682+0000 43'72 45:90 [3,5,1] 3 [3,5,1] 3 0'0 2021-07-19T06:22:23.235056+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.474 INFO:tasks.workunit.client.0.smithi071.stdout:3.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:28.692028+0000 43'67 45:81 [4,3,5] 4 [4,3,5] 4 0'0 2021-07-19T06:22:23.660812+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.475 INFO:tasks.workunit.client.0.smithi071.stdout:3.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:28.693466+0000 43'64 45:78 [0,1,4] 0 [0,1,4] 0 0'0 2021-07-19T06:22:24.062295+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.475 INFO:tasks.workunit.client.0.smithi071.stdout:1.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:24.487000+0000 43'51 44:72 [3,0,1] 3 [3,0,1] 3 0'0 2021-07-26T06:16:22.467639+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.475 INFO:tasks.workunit.client.0.smithi071.stdout:2.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:27.381839+0000 43'53 44:70 [3,0,2] 3 [3,0,2] 3 0'0 2021-07-26T06:16:24.376841+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.475 INFO:tasks.workunit.client.0.smithi071.stdout:3.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:28.691181+0000 43'53 45:67 [4,3,0] 4 [4,3,0] 4 0'0 2021-07-19T06:22:25.276257+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:10.476 INFO:tasks.workunit.client.0.smithi071.stdout:1.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:24.854235+0000 43'73 44:93 [4,2,5] 4 [4,2,5] 4 0'0 2021-07-26T06:16:22.467639+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:10.476 INFO:tasks.workunit.client.0.smithi071.stdout:2.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:26.542414+0000 43'64 45:81 [1,0,4] 1 [1,0,4] 1 0'0 2021-07-19T06:22:26.083082+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:10.476 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:24:10.476 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:24:10.524 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:10.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:24:10.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: grep scrubbing 2021-07-26T06:24:10.621 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:10.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:110: TEST_recover_unexpected: sleep 1 2021-07-26T06:24:11.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: for i in $(seq 0 300) 2021-07-26T06:24:11.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:105: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:24:11.905 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:24:11.916 INFO:tasks.workunit.client.0.smithi071.stdout:3.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:28.690937+0000 43'67 45:82 [3,1,0] 3 [3,1,0] 3 0'0 2021-07-19T06:22:06.851154+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:11.916 INFO:tasks.workunit.client.0.smithi071.stdout:1.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:24.487089+0000 43'62 45:85 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:07.255765+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:11.916 INFO:tasks.workunit.client.0.smithi071.stdout:2.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:26.399489+0000 43'60 45:78 [2,0,5] 2 [2,0,5] 2 0'0 2021-07-19T06:22:07.681429+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:11.923 INFO:tasks.workunit.client.0.smithi071.stdout:3.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:28.694238+0000 43'60 45:73 [5,3,1] 5 [5,3,1] 5 0'0 2021-07-19T06:22:08.086646+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:11.923 INFO:tasks.workunit.client.0.smithi071.stdout:1.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:24.160101+0000 43'69 45:90 [1,4,2] 1 [1,4,2] 1 0'0 2021-07-19T06:22:08.491174+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:11.923 INFO:tasks.workunit.client.0.smithi071.stdout:2.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:26.500677+0000 43'67 45:83 [5,1,3] 5 [5,1,3] 5 0'0 2021-07-19T06:22:08.927807+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:11.923 INFO:tasks.workunit.client.0.smithi071.stdout:3.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:28.693011+0000 43'62 45:77 [0,1,4] 0 [0,1,4] 0 0'0 2021-07-19T06:22:09.332907+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:11.924 INFO:tasks.workunit.client.0.smithi071.stdout:1.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:24.195729+0000 43'67 45:89 [2,4,1] 2 [2,4,1] 2 0'0 2021-07-19T06:22:09.737723+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:11.959 INFO:tasks.workunit.client.0.smithi071.stdout:2.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:26.500699+0000 43'69 45:87 [2,3,5] 2 [2,3,5] 2 0'0 2021-07-19T06:22:10.142339+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:11.959 INFO:tasks.workunit.client.0.smithi071.stdout:3.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2021-07-26T06:16:28.691787+0000 43'69 45:83 [2,4,1] 2 [2,4,1] 2 0'0 2021-07-19T06:22:10.549906+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:11.960 INFO:tasks.workunit.client.0.smithi071.stdout:1.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2021-07-26T06:16:24.621336+0000 43'60 45:83 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:10.952612+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:11.960 INFO:tasks.workunit.client.0.smithi071.stdout:2.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2021-07-26T06:16:26.542415+0000 43'62 45:81 [0,5,4] 0 [0,5,4] 0 0'0 2021-07-19T06:22:11.400753+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:11.960 INFO:tasks.workunit.client.0.smithi071.stdout:3.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:28.693145+0000 43'61 45:76 [4,2,5] 4 [4,2,5] 4 0'0 2021-07-19T06:22:11.805964+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:11.960 INFO:tasks.workunit.client.0.smithi071.stdout:1.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:23.647423+0000 43'65 45:86 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:12.209906+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:11.961 INFO:tasks.workunit.client.0.smithi071.stdout:2.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:26.399792+0000 43'66 45:85 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-19T06:22:12.656716+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:11.961 INFO:tasks.workunit.client.0.smithi071.stdout:3.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:28.695650+0000 43'66 45:81 [4,1,2] 4 [4,1,2] 4 0'0 2021-07-19T06:22:13.063996+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:11.961 INFO:tasks.workunit.client.0.smithi071.stdout:1.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:24.159370+0000 43'61 45:82 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:13.466481+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:11.961 INFO:tasks.workunit.client.0.smithi071.stdout:2.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:27.381838+0000 43'61 45:80 [4,1,0] 4 [4,1,0] 4 0'0 2021-07-19T06:22:13.870023+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:11.961 INFO:tasks.workunit.client.0.smithi071.stdout:3.b 65 0 0 0 0 532480 0 0 65 65 active+clean+scrubbing 2021-07-26T06:24:07.408483+0000 43'65 46:82 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:14.275766+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:11.962 INFO:tasks.workunit.client.0.smithi071.stdout:1.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:24.159919+0000 43'61 45:82 [1,4,5] 1 [1,4,5] 1 0'0 2021-07-19T06:22:14.682271+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:11.962 INFO:tasks.workunit.client.0.smithi071.stdout:2.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:26.324693+0000 43'61 45:78 [1,5,0] 1 [1,5,0] 1 0'0 2021-07-19T06:22:15.086572+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:11.962 INFO:tasks.workunit.client.0.smithi071.stdout:3.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2021-07-26T06:16:28.691424+0000 43'61 45:76 [3,1,5] 3 [3,1,5] 3 0'0 2021-07-19T06:22:15.493274+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:11.979 INFO:tasks.workunit.client.0.smithi071.stdout:1.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2021-07-26T06:16:24.346599+0000 43'66 45:86 [5,4,3] 5 [5,4,3] 5 0'0 2021-07-19T06:22:15.899737+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:11.979 INFO:tasks.workunit.client.0.smithi071.stdout:2.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2021-07-26T06:16:27.349429+0000 43'65 45:84 [3,2,5] 3 [3,2,5] 3 0'0 2021-07-19T06:22:16.315795+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:11.980 INFO:tasks.workunit.client.0.smithi071.stdout:3.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:28.692572+0000 43'73 45:86 [5,3,4] 5 [5,3,4] 5 0'0 2021-07-19T06:22:16.721229+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:11.980 INFO:tasks.workunit.client.0.smithi071.stdout:1.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:23.954687+0000 43'53 45:74 [1,3,0] 1 [1,3,0] 1 0'0 2021-07-19T06:22:17.125434+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:11.980 INFO:tasks.workunit.client.0.smithi071.stdout:2.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:26.399721+0000 43'51 45:68 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:17.542613+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:11.980 INFO:tasks.workunit.client.0.smithi071.stdout:3.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:28.693313+0000 43'51 45:65 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:17.948360+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:11.981 INFO:tasks.workunit.client.0.smithi071.stdout:1.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:24.234706+0000 43'64 45:87 [3,4,2] 3 [3,4,2] 3 0'0 2021-07-19T06:22:18.378551+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:11.981 INFO:tasks.workunit.client.0.smithi071.stdout:2.5 73 0 0 0 0 598016 0 0 73 73 active+clean+scrubbing 2021-07-26T06:24:08.401520+0000 43'73 46:94 [3,4,0] 3 [3,4,0] 3 0'0 2021-07-19T06:22:18.784378+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:11.981 INFO:tasks.workunit.client.0.smithi071.stdout:1.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:24.346071+0000 43'67 45:87 [5,0,4] 5 [5,0,4] 5 0'0 2021-07-19T06:22:19.192427+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:12.009 INFO:tasks.workunit.client.0.smithi071.stdout:2.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:26.357673+0000 43'55 45:71 [5,1,0] 5 [5,1,0] 5 0'0 2021-07-19T06:22:19.598314+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:12.010 INFO:tasks.workunit.client.0.smithi071.stdout:3.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:28.691968+0000 43'54 45:69 [4,0,5] 4 [4,0,5] 4 0'0 2021-07-19T06:22:20.004041+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:12.010 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:23.647564+0000 43'72 45:93 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-19T06:22:20.410121+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:12.010 INFO:tasks.workunit.client.0.smithi071.stdout:2.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:26.542345+0000 43'54 45:70 [5,4,1] 5 [5,4,1] 5 0'0 2021-07-19T06:22:20.813003+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:12.011 INFO:tasks.workunit.client.0.smithi071.stdout:3.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2021-07-26T06:16:28.690784+0000 43'55 45:70 [3,5,0] 3 [3,5,0] 3 0'0 2021-07-19T06:22:21.215259+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:12.011 INFO:tasks.workunit.client.0.smithi071.stdout:1.2 55 0 0 0 0 450560 0 0 55 55 active+clean+scrubbing 2021-07-26T06:24:08.411605+0000 43'55 46:80 [0,4,1] 0 [0,4,1] 0 0'0 2021-07-19T06:22:21.618489+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:12.011 INFO:tasks.workunit.client.0.smithi071.stdout:2.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:26.399501+0000 43'67 45:85 [2,5,0] 2 [2,5,0] 2 0'0 2021-07-19T06:22:22.023080+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:12.011 INFO:tasks.workunit.client.0.smithi071.stdout:3.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:28.692254+0000 43'72 45:86 [1,2,4] 1 [1,2,4] 1 0'0 2021-07-19T06:22:22.429205+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:12.011 INFO:tasks.workunit.client.0.smithi071.stdout:1.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2021-07-26T06:16:24.234654+0000 43'54 45:76 [4,1,2] 4 [4,1,2] 4 0'0 2021-07-19T06:22:22.833352+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:12.012 INFO:tasks.workunit.client.0.smithi071.stdout:2.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2021-07-26T06:16:27.213682+0000 43'72 45:91 [3,5,1] 3 [3,5,1] 3 0'0 2021-07-19T06:22:23.235056+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:12.012 INFO:tasks.workunit.client.0.smithi071.stdout:3.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2021-07-26T06:16:28.692028+0000 43'67 45:82 [4,3,5] 4 [4,3,5] 4 0'0 2021-07-19T06:22:23.660812+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:12.012 INFO:tasks.workunit.client.0.smithi071.stdout:3.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:28.693466+0000 43'64 45:79 [0,1,4] 0 [0,1,4] 0 0'0 2021-07-19T06:22:24.062295+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:12.012 INFO:tasks.workunit.client.0.smithi071.stdout:1.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2021-07-26T06:16:24.487000+0000 43'51 45:74 [3,0,1] 3 [3,0,1] 3 0'0 2021-07-19T06:22:24.466498+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:12.013 INFO:tasks.workunit.client.0.smithi071.stdout:2.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:27.381839+0000 43'53 45:72 [3,0,2] 3 [3,0,2] 3 0'0 2021-07-19T06:22:24.872310+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:12.013 INFO:tasks.workunit.client.0.smithi071.stdout:3.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2021-07-26T06:16:28.691181+0000 43'53 45:68 [4,3,0] 4 [4,3,0] 4 0'0 2021-07-19T06:22:25.276257+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:12.013 INFO:tasks.workunit.client.0.smithi071.stdout:1.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2021-07-26T06:16:24.854235+0000 43'73 45:95 [4,2,5] 4 [4,2,5] 4 0'0 2021-07-19T06:22:25.680858+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:12.013 INFO:tasks.workunit.client.0.smithi071.stdout:2.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2021-07-26T06:16:26.542414+0000 43'64 45:81 [1,0,4] 1 [1,0,4] 1 0'0 2021-07-19T06:22:26.083082+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:12.013 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:24:12.014 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:24:12.029 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:12.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:24:12.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: grep scrubbing 2021-07-26T06:24:12.170 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:24:12.188 INFO:tasks.workunit.client.0.smithi071.stdout:3.b 65 0 0 0 0 532480 0 0 65 65 active+clean+scrubbing 2021-07-26T06:24:07.408483+0000 43'65 46:82 [3,0,4] 3 [3,0,4] 3 0'0 2021-07-19T06:22:14.275766+0000 0'0 2021-07-26T06:16:26.500119+0000 0 2021-07-26T06:24:12.189 INFO:tasks.workunit.client.0.smithi071.stdout:2.5 73 0 0 0 0 598016 0 0 73 73 active+clean+scrubbing 2021-07-26T06:24:08.401520+0000 43'73 46:94 [3,4,0] 3 [3,4,0] 3 0'0 2021-07-19T06:22:18.784378+0000 0'0 2021-07-26T06:16:24.376841+0000 0 2021-07-26T06:24:12.189 INFO:tasks.workunit.client.0.smithi071.stdout:1.2 55 0 0 0 0 450560 0 0 55 55 active+clean+scrubbing 2021-07-26T06:24:08.411605+0000 43'55 46:80 [0,4,1] 0 [0,4,1] 0 0'0 2021-07-19T06:22:21.618489+0000 0'0 2021-07-26T06:16:22.467639+0000 0 2021-07-26T06:24:12.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:107: TEST_recover_unexpected: ok=true 2021-07-26T06:24:12.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:108: TEST_recover_unexpected: break 2021-07-26T06:24:12.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:112: TEST_recover_unexpected: test true = false 2021-07-26T06:24:12.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:117: TEST_recover_unexpected: local total=0 2021-07-26T06:24:12.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:118: TEST_recover_unexpected: local zerocount=0 2021-07-26T06:24:12.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:119: TEST_recover_unexpected: local maxzerocount=3 2021-07-26T06:24:12.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:24:12.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:24:12.208 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:24:12.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:24:12.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:12.209 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:12.209 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:12.209 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:12.209 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:12.210 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:12.210 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:12.210 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:12.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:12.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:12.280 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:12.287 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:12.303 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:24:12.303 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:12.304 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:12.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:12.306 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:12.306 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:12.306 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:12.371 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:12.371 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:12.371 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:12.372 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:12.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:12.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:12.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:12.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:12.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:24:12.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:24:12.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:12.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:12.402 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:12.402 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:12.402 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:12.402 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:12.402 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:12.403 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:12.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:12.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:12.506 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:12.508 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:12.508 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:24:12.508 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:12.617 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:12.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:12.634 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:12.634 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:12.653 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:12.654 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:12.654 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:12.662 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:12.662 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:12.663 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:12.663 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:12.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:12.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:12.665 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2021-07-26T06:24:12.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:24:12.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:12.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:12.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:12.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:12.666 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:12.667 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:12.667 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:12.667 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:12.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:12.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:12.735 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:12.735 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:24:12.735 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:24:12.735 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:12.736 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:12.737 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:12.737 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:12.738 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:12.738 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:12.738 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:12.753 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:12.753 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:12.753 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:12.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:12.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:12.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:12.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:12.816 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2021-07-26T06:24:12.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:24:12.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:12.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:12.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:12.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:12.860 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:12.908 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:12.908 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:12.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:12.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:12.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:12.933 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:12.934 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:24:12.934 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:12.934 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:12.934 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:12.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:12.946 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:12.947 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:12.947 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:12.947 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:12.947 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:12.963 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:12.965 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:12.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:12.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:13.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:13.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:13.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2021-07-26T06:24:13.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:24:13.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:13.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:13.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:13.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:13.098 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:13.098 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:13.098 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:13.129 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:13.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:13.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:13.130 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:13.132 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:13.133 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:24:13.145 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:13.146 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:13.153 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:13.154 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:13.154 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:13.157 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:13.157 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:13.157 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:13.157 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:13.158 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:13.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:13.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:13.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:13.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:13.246 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2021-07-26T06:24:13.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:24:13.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:13.272 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:13.273 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:13.273 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:13.273 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:13.273 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:13.274 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:13.274 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:13.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:13.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:13.333 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:13.333 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:13.333 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:13.334 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:13.334 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:13.385 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:13.386 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:13.386 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:13.386 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:13.387 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:13.387 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:13.387 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:13.396 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:13.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:13.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:13.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:13.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:13.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 18 + 3 2021-07-26T06:24:13.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2021-07-26T06:24:13.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2021-07-26T06:24:13.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:24:13.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 0 + 21 2021-07-26T06:24:13.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=21 2021-07-26T06:24:13.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:24:13.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:24:17.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:24:17.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:24:17.457 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:24:17.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:24:17.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:17.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:17.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:17.522 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:17.524 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:17.524 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:17.575 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:17.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:17.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:17.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:17.576 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:17.577 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:17.577 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:24:17.577 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:17.577 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:17.583 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:17.583 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:17.584 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:17.584 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:17.584 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:17.584 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:17.585 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:17.585 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:17.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:17.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:17.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:17.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:17.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:24:17.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:24:17.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:17.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:17.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:17.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:17.773 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:17.773 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:17.773 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:17.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:17.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:17.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:17.782 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:17.782 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:17.782 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:24:17.783 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:17.789 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:17.794 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:17.794 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:17.795 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:17.795 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:17.795 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:17.795 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:17.795 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:17.796 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:17.796 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:17.796 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:17.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:17.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:17.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2021-07-26T06:24:17.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:24:17.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:17.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:17.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:17.920 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:17.927 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:17.928 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:17.928 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:17.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:17.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:17.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:18.008 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:18.008 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:24:18.008 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:24:18.027 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:18.027 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:18.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:18.036 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:18.036 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:18.037 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:18.073 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:18.086 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:18.086 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:18.087 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:18.088 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:18.088 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:18.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:18.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:18.125 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2021-07-26T06:24:18.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:24:18.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:18.126 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:18.126 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:18.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:18.127 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:18.127 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:18.127 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:18.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:18.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:18.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:18.199 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:18.199 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:24:18.199 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:18.200 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:18.220 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:18.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:18.234 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:18.235 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:18.235 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:18.235 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:18.235 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:18.235 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:18.236 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:18.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:18.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:18.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:18.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:18.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2021-07-26T06:24:18.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:24:18.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:18.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:18.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:18.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:18.317 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:18.318 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:18.319 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:18.319 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:18.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:18.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:18.408 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:18.409 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:18.409 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:24:18.409 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:18.409 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:18.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:18.435 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:18.435 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:18.435 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:18.436 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:18.436 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:18.436 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:18.437 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:18.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:18.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:18.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:18.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:18.531 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2021-07-26T06:24:18.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:24:18.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:18.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:18.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:18.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:18.532 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:18.533 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:18.533 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:18.533 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:18.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:18.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:18.618 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:18.619 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:18.619 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:18.619 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:18.619 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:18.677 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:18.678 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:18.678 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:18.678 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:18.687 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:18.688 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:18.689 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:18.689 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:18.689 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:18.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:18.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:18.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:18.742 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 18 + 3 2021-07-26T06:24:18.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2021-07-26T06:24:18.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2021-07-26T06:24:18.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:24:18.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 21 + 21 2021-07-26T06:24:18.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=42 2021-07-26T06:24:18.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:24:18.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:24:22.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:24:22.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:24:22.741 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:24:22.742 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:24:22.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:22.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:22.747 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:22.747 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:22.747 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:22.747 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:22.748 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:22.748 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:22.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:22.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:22.844 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:22.844 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:22.845 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:24:22.845 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:22.845 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:22.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:22.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:22.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:22.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:22.856 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:22.857 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:22.857 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:22.857 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:22.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:22.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:22.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:22.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:22.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:24:22.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:24:22.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:22.965 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:22.965 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:22.965 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:22.965 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:22.967 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:22.967 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:22.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:22.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:22.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:23.059 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:23.059 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:23.059 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:24:23.060 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:23.060 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:23.066 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:23.066 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:23.066 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:23.067 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:23.070 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:23.070 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:23.070 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:23.071 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:23.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:23.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:23.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:23.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:23.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2021-07-26T06:24:23.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:24:23.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:23.192 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:23.192 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:23.193 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:23.193 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:23.193 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:23.199 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:23.199 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:23.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:23.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:23.266 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:23.266 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:24:23.275 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:24:23.275 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:23.276 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:23.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:23.281 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:23.281 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:23.281 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:23.282 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:23.282 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:23.282 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:23.291 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:23.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:23.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:23.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:23.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:23.382 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2021-07-26T06:24:23.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:24:23.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:23.385 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:23.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:23.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:23.386 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:23.386 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:23.387 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:23.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:23.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:23.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:23.479 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:23.495 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:24:23.497 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:23.497 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:23.497 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:23.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:23.523 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:23.524 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:23.524 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:23.524 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:23.524 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:23.525 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:23.525 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:23.525 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:23.525 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:23.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:23.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:23.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2021-07-26T06:24:23.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:24:23.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:23.644 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:23.645 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:23.645 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:23.645 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:23.645 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:23.646 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:23.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:23.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:23.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:23.686 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:23.686 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:23.687 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:24:23.687 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:23.687 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:23.708 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:23.723 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:23.724 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:23.724 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:23.724 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:23.724 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:23.725 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:23.725 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:23.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:23.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:23.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:23.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:23.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2021-07-26T06:24:23.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:24:23.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:23.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:23.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:23.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:23.809 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:23.810 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:23.810 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:23.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:23.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:23.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:23.895 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:23.895 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:23.896 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:23.896 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:23.896 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:23.934 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:23.934 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:23.943 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:23.944 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:23.944 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:23.944 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:23.944 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:23.960 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:23.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:23.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:24.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:24.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:24.013 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 18 + 3 2021-07-26T06:24:24.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2021-07-26T06:24:24.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2021-07-26T06:24:24.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:24:24.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 42 + 21 2021-07-26T06:24:24.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=63 2021-07-26T06:24:24.015 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:24:24.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:24:28.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:24:28.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:24:28.015 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:24:28.017 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:24:28.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:28.019 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:28.020 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:28.021 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:28.021 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:28.021 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:28.021 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:28.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:28.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:28.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:28.122 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:28.122 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:28.123 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:24:28.123 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:28.123 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:28.209 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:28.210 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:28.210 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:28.211 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:28.211 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:28.212 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:28.212 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:28.212 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:28.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:28.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:28.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:28.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:28.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:24:28.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:24:28.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:28.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:28.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:28.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:28.346 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:28.346 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:28.376 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:28.377 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:28.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:28.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:28.390 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:28.390 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:28.390 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:24:28.390 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:28.391 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:28.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:28.402 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:28.402 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:28.402 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:28.403 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:28.403 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:28.403 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:28.403 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:28.403 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:28.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:28.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:28.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:28.454 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2021-07-26T06:24:28.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:24:28.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:28.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:28.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:28.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:28.463 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:28.463 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:28.464 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:28.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:28.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:28.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:28.547 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:28.547 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:24:28.547 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:24:28.548 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:28.548 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:28.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:28.580 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:28.580 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:28.581 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:28.581 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:28.581 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:28.581 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:28.582 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:28.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:28.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:28.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:28.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:28.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2021-07-26T06:24:28.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:24:28.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:28.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:28.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:28.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:28.670 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:28.671 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:28.671 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:28.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:28.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:28.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:28.756 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:28.762 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:24:28.763 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:28.763 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:28.763 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:28.776 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:28.776 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:28.777 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:28.777 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:28.779 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:28.779 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:28.780 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:28.780 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:28.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:28.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:28.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:28.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:28.870 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2021-07-26T06:24:28.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:24:28.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:28.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:28.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:28.876 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:28.876 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:28.879 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:28.880 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:28.881 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:28.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:28.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:28.966 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:28.967 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:28.967 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:24:28.968 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:28.968 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:28.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:28.994 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:28.994 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:28.994 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:28.994 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:29.019 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:29.021 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:29.021 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:29.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:29.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:29.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:29.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:29.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2021-07-26T06:24:29.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:24:29.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:29.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:29.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:29.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:29.111 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:29.111 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:29.111 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:29.111 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:29.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:29.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:29.178 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:29.179 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:29.179 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:29.179 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:29.179 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:29.199 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:29.200 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:29.200 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:29.200 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:29.200 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:29.201 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:29.201 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:29.201 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:29.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:29.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:29.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:29.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:29.304 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 18 + 3 2021-07-26T06:24:29.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2021-07-26T06:24:29.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2021-07-26T06:24:29.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:24:29.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 63 + 21 2021-07-26T06:24:29.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=84 2021-07-26T06:24:29.306 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:24:29.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:24:33.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:24:33.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:24:33.308 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:24:33.308 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:24:33.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:33.309 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:33.309 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:33.309 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:33.310 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:33.310 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:33.310 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:33.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:33.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:33.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:33.407 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:33.407 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:33.407 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:24:33.408 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:33.408 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:33.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:33.451 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:33.451 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:33.451 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:33.452 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:33.452 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:33.461 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:33.462 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:33.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:33.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:33.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:33.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:33.533 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2021-07-26T06:24:33.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:24:33.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:33.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:33.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:33.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:33.540 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:33.540 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:33.540 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:33.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:33.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:33.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:33.624 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:33.624 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:33.625 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:33.625 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:33.625 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:33.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:33.653 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:33.654 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:33.654 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:33.654 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:33.674 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:33.674 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:33.675 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:33.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:33.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:33.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:24:33.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:24:33.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 2 2021-07-26T06:24:33.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2021-07-26T06:24:33.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:33.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:33.745 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:33.745 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:33.745 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:33.745 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:33.746 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:33.746 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:33.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:33.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:33.829 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:33.829 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:24:33.836 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:33.837 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:33.837 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:33.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:33.843 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:33.844 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:33.844 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:33.844 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:33.844 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:33.845 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:33.845 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:33.845 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:33.845 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:33.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:33.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:33.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 4 2021-07-26T06:24:33.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:24:33.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:33.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:33.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:33.949 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:33.949 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:33.949 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:33.949 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:33.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:33.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:33.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:34.042 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:34.042 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:24:34.042 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:34.054 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:34.054 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:34.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:34.092 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:34.092 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:34.093 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:34.093 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:34.093 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:34.093 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:34.101 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:34.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:34.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:34.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:34.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:34.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 3 2021-07-26T06:24:34.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:24:34.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:34.156 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:34.156 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:34.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:34.157 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:34.157 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:34.158 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:34.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:34.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:34.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:34.260 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:34.261 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:34.271 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:24:34.271 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:34.271 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:34.273 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:34.273 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:34.273 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:34.274 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:34.274 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:34.274 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:34.307 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:34.307 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:34.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:34.308 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:34.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:34.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:34.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 4 2021-07-26T06:24:34.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=16 2021-07-26T06:24:34.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:34.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:34.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:34.430 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:34.430 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:34.430 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:34.431 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:34.431 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:34.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:34.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:34.492 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:34.492 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:34.493 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:24:34.493 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:34.493 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:34.517 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:34.518 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:34.518 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:34.518 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:34.519 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:34.519 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:34.519 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:34.519 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:34.519 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:34.520 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:34.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:34.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:34.595 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 16 + 4 2021-07-26T06:24:34.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=20 2021-07-26T06:24:34.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 20 = 0 ']' 2021-07-26T06:24:34.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:24:34.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 84 + 20 2021-07-26T06:24:34.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=104 2021-07-26T06:24:34.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:24:34.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:24:38.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:24:38.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:24:38.579 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:24:38.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:24:38.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:38.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:38.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:38.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:38.591 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:38.591 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:38.592 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:38.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:38.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:38.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:38.687 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:38.718 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:38.719 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:24:38.744 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:38.745 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:38.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:38.758 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:38.758 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:38.759 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:38.759 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:38.760 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:38.760 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:38.760 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:38.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:38.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:38.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:38.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:38.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:24:38.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:24:38.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:38.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:38.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:38.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:38.809 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:38.809 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:38.809 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:38.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:38.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:38.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:38.899 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:38.899 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:38.908 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:38.909 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:38.909 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:38.920 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:38.920 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:38.933 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:38.934 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:38.934 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:38.934 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:38.934 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:38.935 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:38.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:38.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:39.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:24:39.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:24:39.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2021-07-26T06:24:39.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2021-07-26T06:24:39.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:39.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:39.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:39.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:39.040 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:39.040 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:39.041 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:39.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:39.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:39.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:39.108 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:39.109 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:39.109 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:39.109 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:39.109 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:39.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:39.147 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:39.147 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:39.147 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:39.148 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:39.148 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:39.148 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:39.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:39.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:39.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:39.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:24:39.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:24:39.246 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 2 2021-07-26T06:24:39.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:24:39.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:39.247 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:39.247 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:39.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:39.248 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:39.264 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:39.264 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:39.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:39.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:39.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:39.318 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:39.318 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:39.318 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:39.319 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:39.319 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:39.329 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:39.329 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:39.330 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:39.343 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:39.343 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:39.344 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:39.344 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:39.344 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:39.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:39.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:39.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:39.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:39.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 3 2021-07-26T06:24:39.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:24:39.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:39.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:39.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:39.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:39.444 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:39.444 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:39.444 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:39.445 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:39.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:39.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:39.529 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:39.529 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:39.529 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:39.530 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:39.530 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:39.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:39.555 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:39.556 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:39.556 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:39.556 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:39.556 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:39.557 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:39.557 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:39.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:39.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:39.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:39.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:39.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 3 2021-07-26T06:24:39.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:24:39.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:39.649 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:39.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:39.652 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:39.652 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:39.653 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:39.653 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:39.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:39.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:39.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:39.741 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:39.741 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:24:39.741 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:39.742 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:39.742 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:39.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:39.752 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:39.753 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:39.753 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:39.753 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:39.753 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:39.754 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:39.754 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:39.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:39.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:39.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:39.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:39.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2021-07-26T06:24:39.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:24:39.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2021-07-26T06:24:39.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:24:39.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 104 + 18 2021-07-26T06:24:39.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=122 2021-07-26T06:24:39.864 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:24:39.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:24:43.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:24:43.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:24:43.865 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:24:43.865 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:24:43.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:43.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:43.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:43.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:43.866 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:43.867 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:43.867 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:43.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:43.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:43.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:43.971 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:43.971 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:43.971 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:24:43.972 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:43.972 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:43.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:43.984 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:43.984 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:43.985 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:43.985 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:43.985 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:43.985 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:43.986 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:43.986 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:43.986 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:44.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:44.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:44.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:24:44.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:24:44.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:44.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:44.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:44.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:44.095 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:44.095 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:44.095 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:44.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:44.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:44.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:44.178 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:44.179 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:24:44.179 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:44.179 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:44.180 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:44.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:44.189 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:44.189 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:44.189 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:44.189 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:44.189 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:44.190 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:44.190 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:44.190 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:44.190 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:44.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:44.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:44.292 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2021-07-26T06:24:44.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:24:44.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:44.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:44.295 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:44.295 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:44.295 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:44.296 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:44.296 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:44.296 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:44.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:44.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:44.386 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:44.387 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:44.387 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:44.387 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:44.387 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:44.398 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:44.399 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:44.399 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:44.399 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:44.400 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:44.400 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:44.404 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:44.404 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:44.405 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:44.405 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:44.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:24:44.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:24:44.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2021-07-26T06:24:44.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:24:44.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:44.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:44.502 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:44.502 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:44.502 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:44.502 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:44.503 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:44.503 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:44.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:44.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:44.595 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:44.595 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:44.595 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:44.596 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:44.596 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:44.613 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:44.613 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:44.614 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:44.614 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:44.614 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:44.614 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:44.615 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:44.615 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:44.615 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:44.615 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:44.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:24:44.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:24:44.717 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 2 2021-07-26T06:24:44.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:24:44.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:44.734 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:44.734 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:44.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:44.739 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:44.740 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:44.741 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:44.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:44.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:44.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:44.808 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:44.808 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:44.808 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:24:44.809 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:44.809 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:44.836 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:44.836 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:44.837 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:44.837 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:44.837 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:44.837 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:44.837 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:44.838 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:44.838 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:44.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:44.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:44.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:44.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 4 2021-07-26T06:24:44.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:24:44.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:44.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:44.926 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:44.926 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:44.926 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:44.936 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:44.937 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:44.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:44.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:44.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:45.022 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:45.022 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:24:45.023 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:45.024 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:45.024 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:45.031 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:45.031 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:45.032 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:45.032 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:45.033 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:45.033 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:45.033 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:45.033 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:45.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:45.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:45.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:45.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:45.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 15 + 3 2021-07-26T06:24:45.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:24:45.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2021-07-26T06:24:45.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:24:45.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 122 + 18 2021-07-26T06:24:45.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=140 2021-07-26T06:24:45.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:24:45.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:24:49.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:24:49.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:24:49.135 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:24:49.137 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:24:49.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:49.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:49.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:49.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:49.140 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:49.140 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:49.140 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:49.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:49.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:49.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:49.242 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:49.243 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:49.243 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:24:49.243 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:49.243 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:49.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:49.256 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:49.256 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:49.256 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:49.257 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:49.257 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:49.257 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:49.257 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:49.258 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:49.258 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:49.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:49.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:49.380 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:24:49.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:24:49.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:49.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:49.384 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:49.384 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:49.385 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:49.389 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:49.390 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:49.390 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:49.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:49.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:49.455 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:49.455 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:24:49.455 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:49.456 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:49.456 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:49.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:49.466 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:49.466 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:49.466 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:49.467 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:49.467 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:49.468 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:49.468 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:49.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:49.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:49.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:49.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:49.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2021-07-26T06:24:49.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:24:49.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:49.576 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:49.576 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:49.576 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:49.577 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:49.577 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:49.578 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:49.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:49.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:49.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:49.664 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:49.664 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:49.665 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:49.665 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:49.665 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:49.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:49.675 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:49.675 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:49.675 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:49.675 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:49.676 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:49.676 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:49.676 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:49.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:49.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:49.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:24:49.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:24:49.777 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2021-07-26T06:24:49.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:24:49.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:49.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:49.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:49.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:49.780 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:49.780 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:49.781 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:49.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:49.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:49.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:49.874 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:49.886 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:49.886 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:49.886 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:49.887 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:49.888 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:49.888 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:49.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:49.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:49.889 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:49.889 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:49.890 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:49.890 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:49.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:49.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:49.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:24:49.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:24:49.986 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 2 2021-07-26T06:24:49.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:24:49.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:49.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:49.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:49.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:49.995 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:49.995 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:49.996 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:49.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:49.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:49.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:50.085 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:50.085 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:50.086 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:24:50.086 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:50.086 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:50.114 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:50.115 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:50.115 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:50.116 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:50.116 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:50.125 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:50.125 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:50.126 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:50.126 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:50.126 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:50.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:50.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:50.220 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 4 2021-07-26T06:24:50.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:24:50.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:50.222 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:50.222 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:50.222 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:50.223 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:50.226 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:50.227 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:50.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:50.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:50.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:50.300 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:50.301 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:24:50.301 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:50.302 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:50.302 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:50.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:50.311 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:50.320 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:50.320 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:50.322 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:50.322 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:50.322 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:50.323 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:50.323 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:50.323 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:50.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:50.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:50.409 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 15 + 3 2021-07-26T06:24:50.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:24:50.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2021-07-26T06:24:50.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:24:50.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 140 + 18 2021-07-26T06:24:50.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=158 2021-07-26T06:24:50.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:24:50.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:24:54.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:24:54.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:24:54.429 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:24:54.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:24:54.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:54.430 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:54.430 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:54.430 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:54.431 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:54.431 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:54.434 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:54.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:54.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:54.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:54.520 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:54.520 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:54.521 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:24:54.521 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:54.521 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:54.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:54.536 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:54.537 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:54.537 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:54.537 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:54.537 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:54.537 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:54.538 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:54.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:54.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:54.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:54.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:54.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:24:54.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:24:54.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:54.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:54.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:54.643 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:54.643 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:54.643 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:54.643 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:54.643 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:54.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:54.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:54.732 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:54.744 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:24:54.745 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:54.745 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:54.745 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:54.747 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:54.747 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:54.747 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:54.748 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:54.748 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:54.748 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:54.748 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:54.749 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:54.749 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:54.749 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:24:54.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:54.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:54.845 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2021-07-26T06:24:54.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:24:54.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:54.868 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:54.876 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:54.876 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:54.877 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:54.877 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:54.877 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:54.877 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:54.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:54.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:54.943 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:54.943 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:54.943 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:54.944 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:54.944 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:54.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:54.963 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:24:54.963 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:24:54.963 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:24:54.963 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:54.964 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:54.964 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:54.964 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:24:54.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:54.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:24:55.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:24:55.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:24:55.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2021-07-26T06:24:55.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:24:55.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:55.057 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:55.057 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:55.058 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:55.058 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:55.058 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:55.083 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:55.083 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:55.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:55.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:55.151 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:55.151 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:55.151 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:55.152 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:55.152 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:55.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:55.174 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:24:55.174 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:24:55.174 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:24:55.174 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:55.175 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:55.175 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:55.175 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:24:55.175 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:55.175 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:24:55.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:24:55.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:24:55.280 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 2 2021-07-26T06:24:55.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:24:55.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:55.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:55.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:55.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:55.281 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:55.282 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:55.282 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:55.282 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:55.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:55.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:55.362 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:55.362 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:24:55.363 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:24:55.363 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:55.363 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:55.371 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:55.372 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:24:55.372 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:24:55.372 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:24:55.372 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:55.372 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:55.373 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:55.373 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:24:55.373 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:55.373 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:24:55.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:24:55.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:24:55.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 4 2021-07-26T06:24:55.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:24:55.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:55.475 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:55.476 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:55.476 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:55.491 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:55.491 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:55.492 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:55.492 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:55.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:55.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:55.590 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:55.598 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:24:55.599 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:24:55.599 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:55.599 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:55.627 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:55.628 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:24:55.628 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:24:55.628 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:24:55.628 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:55.629 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:55.629 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:55.629 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:24:55.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:55.630 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:24:55.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:24:55.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:24:55.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 15 + 3 2021-07-26T06:24:55.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:24:55.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2021-07-26T06:24:55.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:24:55.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 158 + 18 2021-07-26T06:24:55.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=176 2021-07-26T06:24:55.707 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:24:55.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:24:59.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:24:59.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:24:59.706 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:24:59.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:24:59.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:59.707 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:59.707 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:59.707 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:59.707 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:59.708 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:59.708 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:59.708 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:59.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:59.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:59.804 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:24:59.805 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:24:59.805 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:24:59.806 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:24:59.806 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:24:59.822 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:24:59.823 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:24:59.823 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:24:59.823 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:24:59.823 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:59.824 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:59.824 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:59.824 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:24:59.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:59.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:24:59.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:24:59.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:24:59.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2021-07-26T06:24:59.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2021-07-26T06:24:59.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:24:59.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:24:59.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:24:59.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:24:59.968 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:24:59.968 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:59.968 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:24:59.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:24:59.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:24:59.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:00.018 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:00.018 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:25:00.020 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:00.020 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:00.020 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:00.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:00.030 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:00.031 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:00.032 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:00.032 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:00.032 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:00.033 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:00.033 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:00.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:00.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:00.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:00.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:00.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 3 2021-07-26T06:25:00.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2021-07-26T06:25:00.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:00.206 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:00.210 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:00.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:00.211 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:00.307 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:00.307 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:00.308 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:00.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:00.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:00.323 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:00.324 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:00.324 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:00.325 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:00.326 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:00.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:00.328 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:00.344 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:00.344 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:00.344 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:00.344 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:00.345 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:00.345 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:00.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:00.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:00.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:25:00.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:25:00.411 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 2 2021-07-26T06:25:00.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:25:00.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:00.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:00.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:00.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:00.415 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:00.415 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:00.415 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:00.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:00.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:00.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:00.440 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:00.441 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:00.447 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:00.447 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:00.447 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:00.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:00.449 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:00.450 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:00.450 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:00.450 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:00.450 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:00.450 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:00.451 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:00.451 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:00.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:00.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:25:00.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:25:00.543 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 1 2021-07-26T06:25:00.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:25:00.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:00.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:00.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:00.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:00.624 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:00.624 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:00.649 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:00.649 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:00.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:00.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:00.650 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:00.651 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:00.651 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:00.651 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:00.651 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:00.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:00.653 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:00.653 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:00.653 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:00.654 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:00.654 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:00.654 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:00.654 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:00.679 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:00.679 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:00.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:00.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:00.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 3 2021-07-26T06:25:00.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:25:00.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:00.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:00.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:00.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:00.757 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:00.757 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:00.757 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:00.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:00.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:00.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:00.851 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:00.872 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:00.872 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:25:00.873 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:00.873 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:00.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:00.887 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:00.915 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:00.915 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:00.915 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:00.916 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:00.927 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:00.927 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:00.927 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:00.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:00.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:25:00.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:25:00.965 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 0 2021-07-26T06:25:00.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:25:00.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 11 = 0 ']' 2021-07-26T06:25:00.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:25:00.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 176 + 11 2021-07-26T06:25:00.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=187 2021-07-26T06:25:00.992 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:25:00.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:25:04.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:25:04.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:25:04.970 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:25:04.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:25:04.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:04.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:05.019 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:05.019 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:05.019 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:05.019 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:05.020 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:05.020 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:05.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:05.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:05.080 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:05.080 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:05.080 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:05.081 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:05.081 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:05.091 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:05.092 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:05.127 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:05.127 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:05.128 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:05.128 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:05.128 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:05.128 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:05.129 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:05.129 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:05.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:25:05.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:25:05.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2021-07-26T06:25:05.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2021-07-26T06:25:05.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:05.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:05.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:05.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:05.242 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:05.242 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:05.268 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:05.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:05.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:05.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:05.299 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:05.299 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:05.299 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:05.300 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:05.300 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:05.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:05.302 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:05.302 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:05.329 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:05.329 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:05.330 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:05.330 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:05.330 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:05.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:05.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:05.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:05.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:05.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 3 2021-07-26T06:25:05.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2021-07-26T06:25:05.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:05.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:05.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:05.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:05.413 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:05.413 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:05.414 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:05.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:05.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:05.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:05.536 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:05.536 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:25:05.537 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:25:05.537 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:05.537 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:05.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:05.577 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:05.577 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:05.577 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:05.590 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:05.590 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:05.590 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:05.591 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:05.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:05.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:05.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:25:05.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:25:05.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 2 2021-07-26T06:25:05.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:25:05.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:05.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:05.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:05.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:05.684 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:05.684 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:05.685 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:05.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:05.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:05.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:05.711 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:05.712 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:05.712 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:05.712 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:05.712 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:05.724 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:05.724 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:05.724 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:05.725 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:05.725 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:05.725 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:05.725 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:05.725 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:05.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:05.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:05.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:05.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:05.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 4 2021-07-26T06:25:05.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:25:05.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:05.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:05.888 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:05.888 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:05.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:05.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:05.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:05.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:05.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:05.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:05.917 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:05.917 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:05.918 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:05.918 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:05.918 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:05.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:05.966 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:05.966 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:05.976 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:05.977 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:05.977 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:05.977 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:05.977 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:05.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:05.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:06.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:06.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:06.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 3 2021-07-26T06:25:06.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:25:06.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:06.030 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:06.045 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:06.045 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:06.063 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:06.063 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:06.063 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:06.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:06.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:06.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:06.131 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:06.131 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:25:06.131 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:06.131 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:06.132 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:06.136 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:06.137 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:06.137 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:06.137 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:06.137 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:06.138 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:06.138 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:06.138 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:06.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:06.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:06.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:06.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:06.249 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2021-07-26T06:25:06.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:25:06.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2021-07-26T06:25:06.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:25:06.274 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 187 + 18 2021-07-26T06:25:06.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=205 2021-07-26T06:25:06.324 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:25:06.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:25:10.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:25:10.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:25:10.255 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:25:10.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:25:10.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:10.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:10.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:10.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:10.257 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:10.257 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:10.257 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:10.257 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:10.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:10.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:10.352 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:10.352 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:10.352 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:10.353 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:10.353 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:10.368 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:10.368 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:10.368 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:10.369 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:10.369 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:10.369 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:10.369 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:10.370 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:10.370 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:10.379 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:10.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:25:10.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:25:10.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2021-07-26T06:25:10.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2021-07-26T06:25:10.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:10.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:10.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:10.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:10.470 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:10.482 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:10.482 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:10.483 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:10.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:10.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:10.566 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:10.566 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:10.566 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:25:10.567 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:10.567 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:10.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:10.586 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:10.597 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:10.597 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:10.597 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:10.597 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:10.598 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:10.598 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:10.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:10.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:10.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:10.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:10.687 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 4 2021-07-26T06:25:10.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2021-07-26T06:25:10.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:10.687 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:10.688 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:10.688 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:10.688 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:10.688 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:10.688 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:10.689 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:10.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:10.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:10.772 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:10.773 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:25:10.773 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:10.773 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:10.774 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:10.793 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:10.794 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:10.794 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:10.794 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:10.794 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:10.795 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:10.795 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:10.795 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:10.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:10.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:10.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:10.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:10.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2021-07-26T06:25:10.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:25:10.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:10.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:10.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:10.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:10.922 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:10.922 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:10.923 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:10.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:10.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:10.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:10.994 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:10.994 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:10.995 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:11.019 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:11.043 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:11.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:11.063 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:11.063 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:11.063 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:11.063 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:11.063 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:11.064 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:11.064 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:11.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:11.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:11.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:11.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:11.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 4 2021-07-26T06:25:11.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2021-07-26T06:25:11.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:11.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:11.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:11.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:11.098 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:11.098 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:11.098 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:11.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:11.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:11.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:11.191 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:11.191 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:11.192 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:11.192 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:11.192 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:11.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:11.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:11.288 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:11.288 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:11.288 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:11.288 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:11.289 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:11.302 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:11.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:11.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:11.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:11.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:11.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 4 2021-07-26T06:25:11.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2021-07-26T06:25:11.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:11.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:11.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:11.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:11.335 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:11.339 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:11.340 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:11.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:11.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:11.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:11.418 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:11.419 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:25:11.419 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:11.420 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:11.420 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:11.421 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:11.421 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:11.422 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:11.422 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:11.422 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:11.426 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:11.426 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:11.427 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:11.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:11.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:11.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:11.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:11.519 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 17 + 4 2021-07-26T06:25:11.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2021-07-26T06:25:11.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2021-07-26T06:25:11.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:25:11.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 205 + 21 2021-07-26T06:25:11.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=226 2021-07-26T06:25:11.524 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:25:11.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:25:15.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:25:15.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:25:15.519 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:25:15.521 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:25:15.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:15.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:15.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:15.524 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:15.524 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:15.524 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:15.524 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:15.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:15.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:15.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:15.629 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:15.630 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:15.630 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:15.630 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:15.630 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:15.644 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:15.645 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:15.645 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:15.645 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:15.645 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:15.646 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:15.646 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:15.646 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:15.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:15.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:15.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:25:15.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:25:15.746 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2021-07-26T06:25:15.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2021-07-26T06:25:15.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:15.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:15.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:15.765 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:15.766 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:15.767 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:15.768 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:15.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:15.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:15.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:15.839 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:15.853 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:15.854 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:25:15.854 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:15.870 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:15.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:15.885 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:15.885 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:15.885 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:15.886 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:15.886 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:15.886 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:15.886 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:15.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:15.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:15.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:15.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:15.953 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 4 2021-07-26T06:25:15.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2021-07-26T06:25:15.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:15.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:15.973 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:15.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:15.974 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:15.974 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:15.974 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:15.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:15.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:15.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:16.051 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:16.051 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:25:16.052 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:16.052 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:16.061 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:16.067 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:16.067 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:16.068 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:16.068 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:16.068 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:16.078 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:16.079 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:16.079 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:16.090 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:16.090 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:16.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:16.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:16.162 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2021-07-26T06:25:16.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:25:16.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:16.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:16.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:16.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:16.185 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:16.185 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:16.185 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:16.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:16.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:16.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:16.259 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:16.259 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:16.259 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:16.260 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:16.260 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:16.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:16.277 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:16.279 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:16.279 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:16.280 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:16.280 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:16.280 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:16.280 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:16.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:16.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:16.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:16.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:16.370 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 4 2021-07-26T06:25:16.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2021-07-26T06:25:16.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:16.371 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:16.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:16.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:16.372 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:16.379 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:16.379 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:16.379 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:16.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:16.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:16.467 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:16.467 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:16.468 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:16.468 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:16.468 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:16.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:16.479 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:16.485 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:16.485 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:16.486 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:16.486 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:16.486 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:16.489 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:16.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:16.490 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:16.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:16.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:16.576 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 4 2021-07-26T06:25:16.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2021-07-26T06:25:16.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:16.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:16.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:16.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:16.582 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:16.583 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:16.583 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:16.583 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:16.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:16.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:16.673 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:16.674 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:25:16.674 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:16.679 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:16.679 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:16.686 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:16.687 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:16.687 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:16.687 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:16.688 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:16.688 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:16.701 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:16.702 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:16.703 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:16.703 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:16.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:16.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:16.788 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 17 + 4 2021-07-26T06:25:16.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2021-07-26T06:25:16.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2021-07-26T06:25:16.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:25:16.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 226 + 21 2021-07-26T06:25:16.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=247 2021-07-26T06:25:16.791 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:25:16.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:25:20.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:25:20.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:25:20.795 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:25:20.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:25:20.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:20.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:20.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:20.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:20.810 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:20.811 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:20.811 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:20.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:20.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:20.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:20.904 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:20.905 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:20.905 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:20.905 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:20.906 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:20.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:20.919 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:20.920 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:20.920 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:20.921 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:20.921 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:20.921 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:20.921 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:20.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:20.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:21.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:25:21.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:25:21.019 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2021-07-26T06:25:21.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2021-07-26T06:25:21.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:21.020 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:21.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:21.051 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:21.052 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:21.059 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:21.060 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:21.061 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:21.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:21.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:21.111 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:21.111 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:21.111 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:25:21.112 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:21.112 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:21.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:21.121 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:21.121 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:21.121 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:21.121 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:21.122 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:21.122 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:21.122 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:21.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:21.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:21.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:21.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:21.223 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 4 2021-07-26T06:25:21.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2021-07-26T06:25:21.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:21.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:21.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:21.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:21.228 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:21.228 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:21.228 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:21.229 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:21.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:21.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:21.321 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:21.322 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:25:21.322 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:21.322 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:21.323 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:21.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:21.331 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:21.331 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:21.331 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:21.332 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:21.332 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:21.332 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:21.333 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:21.333 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:21.333 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:21.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:21.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:21.433 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2021-07-26T06:25:21.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:25:21.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:21.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:21.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:21.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:21.436 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:21.437 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:21.437 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:21.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:21.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:21.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:21.529 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:21.530 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:21.530 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:21.530 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:21.530 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:21.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:21.540 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:21.541 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:21.541 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:21.541 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:21.541 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:21.541 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:21.542 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:21.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:21.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:21.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:21.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:21.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 4 2021-07-26T06:25:21.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2021-07-26T06:25:21.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:21.644 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:21.645 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:21.645 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:21.645 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:21.645 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:21.646 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:21.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:21.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:21.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:21.737 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:21.738 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:21.738 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:21.738 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:21.739 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:21.749 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:21.750 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:21.750 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:21.750 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:21.750 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:21.751 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:21.751 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:21.752 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:21.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:21.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:21.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:21.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:21.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 4 2021-07-26T06:25:21.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2021-07-26T06:25:21.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:21.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:21.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:21.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:21.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:21.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:21.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:21.856 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:21.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:21.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:21.947 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:21.948 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:25:21.948 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:21.948 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:21.949 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:21.959 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:21.960 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:21.960 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:21.960 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:21.961 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:21.961 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:21.962 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:21.962 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:21.963 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:21.963 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:22.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:22.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:22.068 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 17 + 4 2021-07-26T06:25:22.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2021-07-26T06:25:22.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2021-07-26T06:25:22.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:25:22.069 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 247 + 21 2021-07-26T06:25:22.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=268 2021-07-26T06:25:22.069 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:25:22.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:25:26.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:25:26.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:25:26.073 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:25:26.073 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:25:26.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:26.074 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:26.074 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:26.074 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:26.074 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:26.075 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:26.075 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:26.075 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:26.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:26.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:26.176 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:26.177 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:26.177 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:26.177 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:26.178 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:26.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:26.190 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:26.191 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:26.191 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:26.193 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:26.193 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:26.195 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:26.196 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:26.197 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:26.198 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:26.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:25:26.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:25:26.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2021-07-26T06:25:26.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2021-07-26T06:25:26.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:26.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:26.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:26.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:26.294 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:26.294 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:26.294 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:26.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:26.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:26.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:26.387 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:26.388 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:26.389 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:26.389 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:26.389 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:26.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:26.397 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:26.398 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:26.398 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:26.398 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:26.399 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:26.399 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:26.400 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:26.400 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:26.400 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:26.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:26.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:26.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 3 2021-07-26T06:25:26.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:25:26.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:26.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:26.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:26.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:26.505 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:26.506 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:26.506 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:26.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:26.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:26.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:26.597 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:26.598 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:25:26.599 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:26.599 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:26.599 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:26.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:26.607 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:26.608 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:26.609 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:26.609 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:26.610 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:26.610 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:26.610 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:26.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:26.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:26.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:26.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:26.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2021-07-26T06:25:26.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:25:26.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:26.720 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:26.720 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:26.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:26.721 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:26.721 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:26.721 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:26.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:26.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:26.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:26.808 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:26.809 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:26.809 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:26.813 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:26.813 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:26.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:26.826 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:26.826 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:26.827 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:26.827 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:26.827 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:26.828 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:26.828 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:26.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:26.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:26.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:26.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:26.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2021-07-26T06:25:26.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:25:26.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:26.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:26.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:26.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:26.925 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:26.925 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:26.925 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:26.926 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:26.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:26.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:27.019 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:27.019 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:27.020 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:27.020 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:27.020 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:27.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:27.027 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:27.027 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:27.028 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:27.028 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:27.031 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:27.031 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:27.032 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:27.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:27.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:27.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:27.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:27.129 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2021-07-26T06:25:27.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:25:27.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:27.131 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:27.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:27.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:27.133 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:27.133 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:27.133 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:27.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:27.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:27.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:27.224 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:27.225 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:25:27.225 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:27.225 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:27.225 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:27.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:27.235 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:27.235 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:27.236 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:27.236 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:27.236 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:27.236 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:27.237 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:27.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:27.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:27.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:27.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:27.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2021-07-26T06:25:27.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:25:27.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2021-07-26T06:25:27.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:25:27.343 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 268 + 18 2021-07-26T06:25:27.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=286 2021-07-26T06:25:27.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:25:27.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:25:31.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:25:31.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:25:31.348 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:25:31.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:25:31.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:31.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:31.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:31.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:31.357 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:31.357 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:31.361 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:31.362 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:31.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:31.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:31.457 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:31.458 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:31.458 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:25:31.458 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:31.458 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:31.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:31.474 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:31.475 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:31.475 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:31.475 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:31.475 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:31.476 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:31.476 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:31.476 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:31.476 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:31.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:25:31.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:25:31.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:25:31.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:25:31.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:31.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:31.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:31.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:31.585 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:31.585 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:31.585 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:31.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:31.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:31.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:31.668 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:31.668 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:31.669 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:31.671 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:31.671 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:31.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:31.696 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:31.696 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:31.696 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:31.697 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:31.697 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:31.697 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:31.697 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:31.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:31.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:31.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:25:31.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:25:31.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2021-07-26T06:25:31.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2021-07-26T06:25:31.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:31.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:31.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:31.784 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:31.784 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:31.784 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:31.784 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:31.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:31.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:31.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:31.880 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:31.881 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:31.881 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:25:31.881 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:31.881 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:31.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:31.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:31.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:31.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:31.890 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:31.890 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:31.890 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:31.890 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:31.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:31.892 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:32.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:25:32.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:25:32.000 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 1 2021-07-26T06:25:32.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:25:32.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:32.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:32.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:32.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:32.006 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:32.006 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:32.008 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:32.010 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:32.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:32.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:32.089 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:32.089 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:25:32.090 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:32.090 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:32.090 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:32.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:32.100 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:32.100 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:32.101 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:32.101 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:32.101 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:32.101 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:32.102 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:32.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:32.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:32.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:32.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:32.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 3 2021-07-26T06:25:32.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2021-07-26T06:25:32.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:32.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:32.206 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:32.206 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:32.206 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:32.231 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:32.231 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:32.231 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:32.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:32.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:32.303 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:32.303 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:32.304 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:32.304 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:32.305 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:32.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:32.320 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:32.320 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:32.321 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:32.321 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:32.321 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:32.321 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:32.321 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:32.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:32.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:32.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:25:32.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:25:32.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 2 2021-07-26T06:25:32.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:25:32.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:32.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:32.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:32.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:32.432 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:32.433 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:32.434 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:32.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:32.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:32.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:32.508 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:32.508 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:25:32.509 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:32.509 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:32.509 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:32.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:32.518 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:32.519 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:32.519 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:32.519 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:32.520 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:32.520 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:32.520 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:32.520 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:32.521 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:32.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:32.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:32.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 4 2021-07-26T06:25:32.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:25:32.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2021-07-26T06:25:32.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:25:32.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 286 + 12 2021-07-26T06:25:32.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=298 2021-07-26T06:25:32.625 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:25:32.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:25:36.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:25:36.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:25:36.631 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:25:36.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:25:36.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:36.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:36.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:36.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:36.636 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:36.636 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:36.637 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:36.637 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:36.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:36.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:36.737 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:36.737 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:36.737 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:25:36.737 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:36.738 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:36.749 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:36.750 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:36.750 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:36.751 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:36.751 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:36.756 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:36.757 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:36.757 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:36.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:36.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:36.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:25:36.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:25:36.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2021-07-26T06:25:36.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2021-07-26T06:25:36.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:36.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:36.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:36.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:36.859 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:36.859 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:36.859 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:36.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:36.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:36.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:36.950 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:36.950 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:36.951 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:36.951 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:36.951 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:36.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:36.961 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:36.961 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:36.962 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:36.975 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:36.976 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:36.976 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:36.976 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:36.976 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:36.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:37.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:37.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:37.067 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 4 2021-07-26T06:25:37.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2021-07-26T06:25:37.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:37.067 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:37.068 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:37.072 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:37.072 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:37.072 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:37.072 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:37.073 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:37.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:37.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:37.160 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:37.160 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:37.160 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:37.161 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:37.161 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:37.171 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:37.171 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:37.172 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:37.172 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:37.172 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:37.172 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:37.172 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:37.173 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:37.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:37.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:37.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:25:37.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:25:37.274 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 1 2021-07-26T06:25:37.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2021-07-26T06:25:37.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:37.283 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:37.283 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:37.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:37.284 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:37.284 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:37.284 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:37.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:37.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:37.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:37.371 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:37.372 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:25:37.373 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:25:37.374 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:37.374 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:37.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:37.386 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:37.387 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:37.387 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:37.402 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:37.403 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:37.403 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:37.403 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:37.422 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:37.424 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:37.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:25:37.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:25:37.485 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 2 2021-07-26T06:25:37.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:25:37.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:37.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:37.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:37.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:37.506 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:37.517 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:37.517 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:37.517 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:37.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:37.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:37.582 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:37.583 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:37.583 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:37.583 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:37.583 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:37.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:37.605 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:37.609 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:37.609 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:37.610 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:37.610 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:37.610 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:37.610 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:37.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:37.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:37.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:37.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:37.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 3 2021-07-26T06:25:37.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:25:37.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:37.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:37.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:37.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:37.701 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:37.701 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:37.702 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:37.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:37.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:37.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:37.791 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:37.791 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:37.792 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:37.792 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:37.792 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:37.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:37.823 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:37.824 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:37.824 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:37.824 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:37.825 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:37.825 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:37.826 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:37.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:37.856 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:37.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:37.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:37.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 4 2021-07-26T06:25:37.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:25:37.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2021-07-26T06:25:37.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:25:37.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 298 + 15 2021-07-26T06:25:37.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=313 2021-07-26T06:25:37.910 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:25:37.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:25:41.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:25:41.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:25:41.915 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:25:41.915 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:25:41.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:41.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:41.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:41.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:41.919 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:41.920 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:41.920 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:41.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:41.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:41.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:42.023 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:42.023 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:42.023 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:42.024 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:42.024 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:42.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:42.045 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:42.045 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:42.045 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:42.045 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:42.045 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:42.046 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:42.046 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:42.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:42.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:42.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:42.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:42.142 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2021-07-26T06:25:42.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:25:42.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:42.144 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:42.144 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:42.144 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:42.147 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:42.147 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:42.147 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:42.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:42.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:42.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:42.235 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:42.239 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:42.239 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:42.239 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:42.240 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:42.242 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:42.242 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:42.242 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:42.243 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:42.243 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:42.243 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:42.243 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:42.243 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:42.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:42.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:42.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:42.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:42.343 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2021-07-26T06:25:42.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:25:42.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:42.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:42.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:42.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:42.346 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:42.346 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:42.346 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:42.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:42.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:42.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:42.439 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:42.439 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:42.439 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:42.440 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:42.440 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:42.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:42.452 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:42.452 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:42.452 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:42.453 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:42.453 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:42.453 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:42.453 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:42.453 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:42.454 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:42.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:25:42.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:25:42.552 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2021-07-26T06:25:42.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:25:42.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:42.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:42.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:42.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:42.555 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:42.555 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:42.556 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:42.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:42.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:42.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:42.647 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:42.647 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 4, 2021-07-26T06:25:42.648 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:25:42.648 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:42.648 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:42.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:42.658 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:42.658 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:42.659 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:42.659 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:42.659 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:42.659 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:42.660 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:42.660 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:42.660 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:42.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:42.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:42.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 4 2021-07-26T06:25:42.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2021-07-26T06:25:42.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:42.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:42.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:42.764 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:42.764 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:42.764 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:42.764 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:42.765 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:42.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:42.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:42.858 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:42.858 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:42.858 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:25:42.858 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:42.859 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:42.871 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:42.872 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:42.873 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:42.873 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:42.874 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:42.874 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:42.874 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:42.874 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:42.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:42.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:42.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:42.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:42.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 4 2021-07-26T06:25:42.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2021-07-26T06:25:42.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:42.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:42.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:42.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:42.977 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:42.978 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:42.978 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:42.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:42.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:42.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:43.069 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:43.070 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:43.070 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:43.070 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:43.071 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:43.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:43.080 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:43.081 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:43.081 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:43.081 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:43.081 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:43.082 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:43.082 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:43.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:43.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:43.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:43.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:43.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 17 + 4 2021-07-26T06:25:43.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2021-07-26T06:25:43.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2021-07-26T06:25:43.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:25:43.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 313 + 21 2021-07-26T06:25:43.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=334 2021-07-26T06:25:43.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:25:43.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:25:47.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:25:47.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:25:47.191 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:25:47.192 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:25:47.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:47.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:47.197 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:47.197 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:47.198 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:47.203 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:47.204 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:47.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:47.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:47.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:47.298 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:47.298 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:47.299 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:47.299 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:47.299 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:47.308 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:47.309 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:47.309 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:47.310 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:47.311 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:47.311 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:47.312 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:47.313 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:47.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:47.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:47.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:47.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:47.420 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2021-07-26T06:25:47.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:25:47.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:47.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:47.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:47.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:47.437 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:47.437 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:47.437 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:47.438 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:47.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:47.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:47.514 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:47.515 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:47.515 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:47.515 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:47.515 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:47.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:47.529 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:47.529 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:47.530 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:47.530 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:47.555 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:47.555 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:47.556 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:47.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:47.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:47.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:47.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:47.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2021-07-26T06:25:47.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:25:47.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:47.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:47.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:47.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:47.637 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:47.637 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:47.638 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:47.638 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:47.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:47.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:47.722 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:47.723 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:47.725 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:47.726 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:47.728 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:47.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:47.731 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:47.731 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:47.732 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:47.732 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:47.732 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:47.732 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:47.733 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:47.733 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:47.733 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:47.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:25:47.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:25:47.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2021-07-26T06:25:47.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:25:47.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:47.837 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:47.838 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:47.838 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:47.838 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:47.838 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:47.839 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:47.839 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:47.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:47.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:47.932 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:47.932 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 4, 2021-07-26T06:25:47.933 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:25:47.933 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:47.933 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:47.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:47.941 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:47.941 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:47.942 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:47.943 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:47.943 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:47.944 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:47.944 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:47.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:47.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:48.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:48.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:48.043 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 4 2021-07-26T06:25:48.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2021-07-26T06:25:48.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:48.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:48.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:48.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:48.047 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:48.047 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:48.047 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:48.048 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:48.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:48.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:48.149 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:48.160 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:48.160 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:25:48.161 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:48.161 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:48.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:48.166 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:48.166 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:48.166 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:48.166 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:48.167 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:48.167 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:48.167 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:48.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:48.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:48.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:48.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:48.253 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 4 2021-07-26T06:25:48.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2021-07-26T06:25:48.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:48.255 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:48.258 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:48.259 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:48.259 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:48.260 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:48.260 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:48.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:48.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:48.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:48.349 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:48.349 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:48.349 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:48.350 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:48.350 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:48.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:48.372 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:48.372 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:48.373 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:48.375 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:48.376 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:48.376 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:48.376 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:48.376 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:48.377 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:48.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:48.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:48.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 17 + 4 2021-07-26T06:25:48.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2021-07-26T06:25:48.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2021-07-26T06:25:48.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:25:48.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 334 + 21 2021-07-26T06:25:48.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=355 2021-07-26T06:25:48.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:25:48.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:25:52.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:25:52.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:25:52.472 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:25:52.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:25:52.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:52.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:52.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:52.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:52.483 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:52.484 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:52.484 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:52.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:52.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:52.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:52.578 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:52.579 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:52.579 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:52.579 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:52.580 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:52.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:52.589 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:52.590 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:52.590 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:52.590 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:52.590 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:52.591 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:52.591 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:52.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:52.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:52.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:52.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:52.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:25:52.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:25:52.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:52.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:52.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:52.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:52.697 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:52.697 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:52.697 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:52.703 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:52.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:52.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:52.793 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:52.793 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:25:52.793 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:52.794 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:52.794 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:52.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:52.801 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:52.802 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:52.802 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:52.802 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:52.803 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:52.803 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:52.804 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:52.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:52.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:52.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:52.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:52.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2021-07-26T06:25:52.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:25:52.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:52.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:52.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:52.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:52.913 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:52.913 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:52.914 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:52.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:52.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:52.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:53.002 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:53.003 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:53.003 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:53.003 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:53.003 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:53.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:53.012 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:53.013 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:53.013 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:53.013 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:53.013 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:53.014 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:53.014 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:53.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:53.015 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:53.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:25:53.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:25:53.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 2 2021-07-26T06:25:53.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:25:53.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:53.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:53.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:53.124 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:53.126 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:53.126 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:53.126 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:53.126 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:53.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:53.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:53.211 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:53.211 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 4, 2021-07-26T06:25:53.212 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:25:53.212 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:53.212 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:53.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:53.234 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:53.234 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:53.235 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:53.235 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:53.235 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:53.236 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:53.236 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:53.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:53.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:53.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:53.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:53.329 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2021-07-26T06:25:53.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:25:53.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:53.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:53.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:53.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:53.330 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:53.331 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:53.331 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:53.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:53.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:53.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:53.422 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:53.422 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:53.423 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:25:53.423 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:53.423 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:53.433 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:53.433 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:53.434 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:53.434 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:53.434 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:53.434 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:53.435 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:53.435 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:53.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:53.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:53.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:53.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:53.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2021-07-26T06:25:53.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:25:53.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:53.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:53.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:53.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:53.540 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:53.540 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:53.540 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:53.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:53.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:53.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:53.634 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:53.635 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:53.635 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:53.635 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:53.635 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:53.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:53.651 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:53.652 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:53.652 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:53.652 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:53.653 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:53.653 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:53.653 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:53.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:53.654 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:53.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:53.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:53.747 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 18 + 3 2021-07-26T06:25:53.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2021-07-26T06:25:53.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2021-07-26T06:25:53.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:25:53.749 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 355 + 21 2021-07-26T06:25:53.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=376 2021-07-26T06:25:53.750 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:25:53.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:25:57.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:25:57.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:25:57.750 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:25:57.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:25:57.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:57.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:57.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:57.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:57.756 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:57.756 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:57.757 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:57.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:57.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:57.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:57.858 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:57.859 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:25:57.859 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:25:57.859 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:57.859 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:57.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:57.872 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:25:57.872 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:25:57.873 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:25:57.873 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:57.874 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:57.874 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:57.875 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:25:57.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:57.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:25:57.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:57.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:57.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:25:57.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:25:57.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:57.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:57.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:57.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:57.977 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:57.978 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:57.978 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:57.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:57.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:57.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:58.071 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:58.071 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:25:58.072 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:58.072 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:58.073 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:58.081 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:58.082 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:25:58.082 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:25:58.082 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:25:58.082 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:58.083 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:58.083 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:58.083 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:25:58.083 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:58.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:25:58.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:58.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:58.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2021-07-26T06:25:58.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:25:58.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:58.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:58.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:58.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:58.188 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:58.188 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:58.188 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:58.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:58.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:58.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:58.283 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:58.284 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:58.284 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:25:58.284 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:58.286 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:58.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:58.290 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:25:58.295 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:25:58.295 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:25:58.295 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:58.295 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:58.296 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:58.296 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:25:58.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:58.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:25:58.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:25:58.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:25:58.398 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 1 2021-07-26T06:25:58.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:25:58.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:58.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:58.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:58.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:58.399 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:58.400 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:58.400 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:58.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:58.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:58.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:58.491 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:58.491 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:25:58.491 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:25:58.492 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:58.492 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:58.503 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:58.503 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:25:58.504 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:25:58.504 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:25:58.504 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:58.504 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:58.505 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:58.505 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:25:58.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:58.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:25:58.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:25:58.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:25:58.601 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 3 2021-07-26T06:25:58.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:25:58.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:58.603 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:58.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:58.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:58.604 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:58.604 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:58.605 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:58.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:58.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:58.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:58.698 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:58.698 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:58.698 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:25:58.699 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:58.699 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:58.709 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:58.710 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:25:58.710 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:25:58.710 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:25:58.710 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:58.710 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:58.711 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:58.711 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:25:58.711 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:58.711 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:25:58.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:25:58.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:25:58.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 4 2021-07-26T06:25:58.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=16 2021-07-26T06:25:58.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:25:58.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:58.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:58.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:58.813 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:58.813 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:58.813 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:58.814 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:58.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:58.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:58.911 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:25:58.912 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:25:58.912 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:25:58.912 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:25:58.913 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:25:58.920 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:25:58.921 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:25:58.921 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:25:58.922 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:25:58.922 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:25:58.922 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:25:58.923 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:25:58.923 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:25:58.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:25:58.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:25:59.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:25:59.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:25:59.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 16 + 1 2021-07-26T06:25:59.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2021-07-26T06:25:59.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 17 = 0 ']' 2021-07-26T06:25:59.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:25:59.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 376 + 17 2021-07-26T06:25:59.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=393 2021-07-26T06:25:59.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:25:59.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:26:03.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:26:03.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:26:03.029 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:26:03.030 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:26:03.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:03.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:03.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:03.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:03.034 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:03.034 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:03.035 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:03.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:03.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:03.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:03.137 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:03.138 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:03.138 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:03.138 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:03.139 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:03.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:03.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:03.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:03.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:03.148 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:03.149 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:03.149 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:03.149 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:03.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:03.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:03.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:03.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:03.251 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2021-07-26T06:26:03.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:26:03.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:03.254 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:03.254 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:03.254 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:03.255 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:03.255 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:03.256 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:03.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:03.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:03.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:03.348 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:03.349 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:03.349 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:26:03.349 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:03.349 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:03.357 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:03.358 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:03.358 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:03.359 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:03.359 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:03.359 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:03.359 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:03.360 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:03.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:03.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:03.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:26:03.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:26:03.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 1 2021-07-26T06:26:03.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:26:03.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:03.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:03.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:03.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:03.466 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:03.466 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:03.466 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:03.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:03.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:03.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:03.559 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:03.559 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:03.559 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:26:03.559 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:03.560 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:03.569 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:03.569 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:03.569 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:03.570 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:03.570 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:03.570 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:03.570 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:03.571 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:03.571 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:03.571 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:03.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:26:03.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:26:03.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 0 2021-07-26T06:26:03.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:26:03.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:03.673 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:03.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:03.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:03.674 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:03.674 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:03.675 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:03.675 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:03.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:03.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:03.769 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:03.769 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:03.769 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:03.770 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:03.770 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:03.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:03.780 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:03.780 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:03.781 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:03.781 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:03.781 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:03.781 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:03.781 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:03.782 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:03.782 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:03.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:26:03.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:26:03.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2021-07-26T06:26:03.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2021-07-26T06:26:03.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:03.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:03.886 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:03.886 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:03.886 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:03.886 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:03.886 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:03.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:03.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:03.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:03.979 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:03.979 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:03.979 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:03.980 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:03.980 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:03.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:03.990 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:03.990 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:03.991 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:03.991 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:03.992 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:03.992 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:03.992 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:03.992 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:03.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:04.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:04.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:04.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 4 2021-07-26T06:26:04.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:26:04.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:04.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:04.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:04.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:04.095 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:04.096 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:04.096 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:04.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:04.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:04.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:04.188 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:04.188 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:04.189 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:04.189 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:04.189 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:04.198 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:04.199 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:04.200 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:04.200 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:04.200 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:04.200 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:04.200 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:04.201 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:04.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:04.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:04.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:26:04.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:26:04.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 2 2021-07-26T06:26:04.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:26:04.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2021-07-26T06:26:04.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:26:04.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 393 + 12 2021-07-26T06:26:04.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=405 2021-07-26T06:26:04.304 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:26:04.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:26:08.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:26:08.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:26:08.307 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:26:08.309 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:26:08.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:08.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:08.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:08.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:08.311 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:08.312 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:08.312 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:08.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:08.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:08.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:08.415 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:08.415 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:08.416 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:08.416 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:08.416 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:08.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:08.426 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:08.427 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:08.427 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:08.434 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:08.434 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:08.434 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:08.434 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:08.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:08.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:08.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:26:08.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:26:08.530 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2021-07-26T06:26:08.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2021-07-26T06:26:08.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:08.533 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:08.533 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:08.533 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:08.533 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:08.534 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:08.534 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:08.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:08.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:08.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:08.627 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:08.627 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:08.628 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:08.628 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:08.628 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:08.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:08.640 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:08.640 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:08.641 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:08.641 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:08.641 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:08.641 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:08.642 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:08.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:08.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:08.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:08.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:08.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 3 2021-07-26T06:26:08.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2021-07-26T06:26:08.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:08.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:08.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:08.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:08.744 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:08.744 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:08.745 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:08.745 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:08.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:08.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:08.836 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:08.837 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:08.837 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:08.837 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:08.839 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:08.849 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:08.849 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:08.849 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:08.850 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:08.850 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:08.850 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:08.850 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:08.851 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:08.851 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:08.851 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:08.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:26:08.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:26:08.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 2 2021-07-26T06:26:08.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:26:08.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:08.953 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:08.953 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:08.953 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:08.953 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:08.954 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:08.954 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:08.954 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:08.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:08.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:09.048 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:09.048 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:26:09.050 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:09.050 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:09.050 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:09.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:09.064 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:09.065 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:09.065 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:09.065 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:09.075 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:09.075 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:09.075 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:09.076 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:09.076 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:09.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:09.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:09.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 4 2021-07-26T06:26:09.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:26:09.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:09.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:09.169 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:09.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:09.170 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:09.172 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:09.172 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:09.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:09.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:09.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:09.257 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:09.258 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:09.258 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:09.258 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:09.258 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:09.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:09.268 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:09.268 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:09.268 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:09.268 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:09.269 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:09.269 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:09.269 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:09.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:09.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:09.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:09.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:09.376 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 4 2021-07-26T06:26:09.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:26:09.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:09.377 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:09.377 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:09.378 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:09.378 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:09.378 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:09.379 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:09.379 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:09.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:09.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:09.467 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:09.467 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:09.467 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:09.468 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:09.468 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:09.476 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:09.476 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:09.477 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:09.477 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:09.477 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:09.478 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:09.478 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:09.478 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:09.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:09.479 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:09.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:09.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:09.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 15 + 3 2021-07-26T06:26:09.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:26:09.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2021-07-26T06:26:09.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:26:09.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 405 + 18 2021-07-26T06:26:09.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=423 2021-07-26T06:26:09.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:26:09.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:26:13.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:26:13.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:26:13.593 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:26:13.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:26:13.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:13.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:13.595 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:13.595 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:13.595 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:13.595 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:13.595 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:13.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:13.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:13.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:13.693 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:13.694 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:26:13.694 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:13.694 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:13.695 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:13.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:13.707 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:13.707 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:13.707 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:13.707 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:13.708 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:13.708 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:13.708 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:13.708 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:13.708 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:13.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:13.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:13.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:26:13.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:26:13.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:13.814 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:13.814 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:13.814 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:13.815 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:13.815 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:13.815 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:13.815 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:13.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:13.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:13.916 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:13.916 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:13.916 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:13.917 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:13.917 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:13.926 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:13.926 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:13.926 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:13.926 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:13.927 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:13.927 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:13.927 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:13.927 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:13.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:13.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:14.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:14.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:14.016 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2021-07-26T06:26:14.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:26:14.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:14.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:14.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:14.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:14.026 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:14.026 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:14.026 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:14.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:14.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:14.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:14.112 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:14.113 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:14.113 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:14.113 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:14.113 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:14.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:14.135 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:14.137 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:14.138 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:14.138 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:14.138 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:14.139 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:14.139 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:14.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:14.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:14.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:26:14.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:26:14.231 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 2 2021-07-26T06:26:14.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:26:14.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:14.231 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:14.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:14.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:14.237 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:14.237 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:14.238 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:14.238 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:14.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:14.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:14.322 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:14.323 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:26:14.323 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:14.323 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:14.323 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:14.338 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:14.340 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:14.340 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:14.341 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:14.341 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:14.341 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:14.341 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:14.342 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:14.342 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:14.342 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:14.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:14.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:14.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2021-07-26T06:26:14.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:26:14.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:14.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:14.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:14.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:14.443 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:14.443 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:14.443 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:14.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:14.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:14.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:14.536 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:14.537 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:14.537 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:14.538 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:14.542 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:14.547 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:14.547 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:14.548 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:14.548 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:14.548 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:14.549 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:14.549 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:14.549 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:14.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:14.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:14.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:14.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:14.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2021-07-26T06:26:14.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:26:14.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:14.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:14.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:14.649 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:14.649 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:14.649 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:14.649 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:14.649 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:14.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:14.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:14.743 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:14.744 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:14.744 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:14.744 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:14.745 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:14.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:14.775 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:14.776 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:14.776 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:14.776 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:14.776 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:14.777 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:14.777 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:14.777 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:14.777 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:14.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:14.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:14.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 18 + 3 2021-07-26T06:26:14.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2021-07-26T06:26:14.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2021-07-26T06:26:14.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:26:14.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 423 + 21 2021-07-26T06:26:14.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=444 2021-07-26T06:26:14.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:26:14.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:26:18.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:26:18.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:26:18.864 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:26:18.864 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:26:18.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:18.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:18.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:18.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:18.867 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:18.868 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:18.868 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:18.868 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:18.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:18.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:18.972 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:18.972 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:26:18.973 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:18.973 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:18.973 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:18.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:18.984 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:18.984 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:18.984 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:18.984 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:18.985 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:18.985 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:18.985 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:18.985 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:18.986 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:19.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:19.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:19.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2021-07-26T06:26:19.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:26:19.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:19.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:19.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:19.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:19.095 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:19.095 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:19.095 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:19.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:19.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:19.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:19.183 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:19.183 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:19.183 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:19.183 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:19.184 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:19.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:19.218 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:19.219 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:19.219 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:19.219 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:19.220 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:19.220 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:19.220 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:19.220 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:19.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:19.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:19.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:19.296 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2021-07-26T06:26:19.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:26:19.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:19.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:19.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:19.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:19.300 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:19.301 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:19.301 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:19.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:19.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:19.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:19.391 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:19.392 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:19.393 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:19.393 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:19.394 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:19.403 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:19.403 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:19.403 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:19.404 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:19.404 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:19.404 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:19.406 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:19.406 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:19.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:19.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:19.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:19.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:19.509 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2021-07-26T06:26:19.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:26:19.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:19.509 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:19.509 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:19.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:19.510 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:19.510 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:19.510 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:19.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:19.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:19.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:19.605 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:19.607 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:26:19.607 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:19.607 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:19.607 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:19.625 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:19.625 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:19.625 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:19.626 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:19.637 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:19.638 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:19.638 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:19.639 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:19.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:19.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:19.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:19.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:19.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2021-07-26T06:26:19.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:26:19.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:19.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:19.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:19.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:19.727 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:19.727 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:19.727 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:19.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:19.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:19.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:19.811 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:19.812 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:19.812 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:19.813 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:19.813 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:19.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:19.824 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:19.824 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:19.830 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:19.831 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:19.831 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:19.831 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:19.831 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:19.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:19.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:19.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:19.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:19.927 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2021-07-26T06:26:19.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:26:19.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:19.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:19.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:19.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:19.936 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:19.936 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:19.936 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:19.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:19.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:19.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:20.025 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:20.025 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:20.026 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:20.026 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:20.027 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:20.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:20.032 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:20.033 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:20.033 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:20.033 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:20.033 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:20.034 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:20.034 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:20.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:20.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:20.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:20.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:20.138 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 18 + 3 2021-07-26T06:26:20.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2021-07-26T06:26:20.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2021-07-26T06:26:20.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:26:20.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 444 + 21 2021-07-26T06:26:20.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=465 2021-07-26T06:26:20.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:26:20.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:26:24.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:26:24.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:26:24.145 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:26:24.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:26:24.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:24.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:24.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:24.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:24.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:24.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:24.149 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:24.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:24.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:24.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:24.249 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:24.249 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:26:24.250 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:24.250 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:24.250 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:24.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:24.261 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:24.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:24.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:24.262 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:24.262 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:24.263 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:24.263 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:24.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:24.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:24.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:24.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:24.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2021-07-26T06:26:24.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:26:24.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:24.373 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:24.374 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:24.375 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:24.376 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:24.376 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:24.378 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:24.378 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:24.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:24.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:24.460 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:24.464 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:24.468 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:24.471 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:24.472 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:24.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:24.487 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:24.489 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:24.489 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:24.504 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:24.509 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:24.509 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:24.510 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:24.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:24.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:24.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:24.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:24.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2021-07-26T06:26:24.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:26:24.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:24.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:24.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:24.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:24.580 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:24.580 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:24.581 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:24.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:24.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:24.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:24.669 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:24.670 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:24.670 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:24.671 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:24.671 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:24.686 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:24.695 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:24.695 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:24.695 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:24.705 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:24.705 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:24.706 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:24.706 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:24.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:24.709 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:24.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:24.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:24.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2021-07-26T06:26:24.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:26:24.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:24.790 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:24.790 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:24.790 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:24.795 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:24.796 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:24.796 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:24.796 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:24.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:24.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:24.879 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:24.880 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:26:24.880 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:24.880 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:24.880 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:24.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:24.890 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:24.891 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:24.891 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:24.891 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:24.892 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:24.902 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:24.902 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:24.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:24.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:24.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:24.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:24.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2021-07-26T06:26:24.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:26:24.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:24.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:24.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:24.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:24.998 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:24.998 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:24.999 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:24.999 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:24.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:25.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:25.091 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:25.091 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:25.091 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:25.092 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:25.092 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:25.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:25.102 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:25.103 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:25.103 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:25.103 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:25.103 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:25.103 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:25.104 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:25.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:25.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:25.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:25.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:25.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2021-07-26T06:26:25.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:26:25.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:25.214 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:25.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:25.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:25.235 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:25.236 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:25.236 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:25.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:25.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:25.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:25.302 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:25.302 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:25.303 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:25.303 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:25.304 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:25.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:25.314 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:25.315 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:25.315 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:25.316 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:25.316 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:25.316 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:25.317 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:25.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:25.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:25.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:25.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:25.417 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 18 + 3 2021-07-26T06:26:25.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2021-07-26T06:26:25.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2021-07-26T06:26:25.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:26:25.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 465 + 21 2021-07-26T06:26:25.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=486 2021-07-26T06:26:25.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:26:25.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:26:29.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:26:29.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:26:29.422 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:26:29.424 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:26:29.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:29.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:29.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:29.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:29.437 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:29.438 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:29.448 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:29.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:29.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:29.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:29.530 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:29.530 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:29.531 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:29.532 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:29.532 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:29.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:29.550 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:29.550 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:29.551 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:29.551 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:29.551 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:29.551 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:29.552 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:29.553 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:29.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:29.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:29.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:29.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2021-07-26T06:26:29.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:26:29.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:29.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:29.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:29.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:29.651 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:29.651 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:29.652 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:29.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:29.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:29.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:29.743 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:29.751 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:29.751 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:29.751 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:29.752 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:29.784 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:29.784 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:29.784 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:29.784 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:29.787 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:29.787 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:29.788 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:29.788 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:29.788 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:29.788 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:29.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:29.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:29.871 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2021-07-26T06:26:29.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:26:29.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:29.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:29.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:29.886 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:29.886 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:29.891 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:29.891 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:29.892 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:29.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:29.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:29.955 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:29.956 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:29.956 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:29.956 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:29.957 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:29.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:29.991 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:29.996 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:29.997 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:29.997 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:29.997 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:29.997 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:29.997 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:29.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:29.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:30.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:30.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:30.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2021-07-26T06:26:30.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:26:30.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:30.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:30.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:30.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:30.111 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:30.112 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:30.112 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:30.113 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:30.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:30.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:30.162 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:30.163 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 4, 2021-07-26T06:26:30.163 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:26:30.164 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:30.164 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:30.181 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:30.181 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:30.182 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:30.184 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:30.185 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:30.185 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:30.185 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:30.186 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:30.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:30.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:30.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:30.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:30.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2021-07-26T06:26:30.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:26:30.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:30.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:30.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:30.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:30.279 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:30.280 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:30.280 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:30.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:30.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:30.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:30.367 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:30.367 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:30.368 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:30.368 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:30.368 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:30.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:30.381 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:30.382 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:30.383 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:30.383 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:30.383 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:30.383 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:30.384 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:30.384 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:30.384 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:30.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:30.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:30.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 3 2021-07-26T06:26:30.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2021-07-26T06:26:30.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:30.490 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:30.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:30.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:30.498 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:30.499 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:30.500 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:30.500 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:30.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:30.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:30.581 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:30.581 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:30.582 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:30.582 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:30.588 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:30.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:30.591 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:30.592 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:30.592 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:30.593 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:30.602 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:30.615 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:30.615 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:30.615 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:30.616 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:30.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:26:30.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:26:30.695 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 17 + 1 2021-07-26T06:26:30.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:26:30.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2021-07-26T06:26:30.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:26:30.697 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 486 + 18 2021-07-26T06:26:30.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=504 2021-07-26T06:26:30.697 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:26:30.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:26:34.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:26:34.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:26:34.735 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:26:34.736 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:26:34.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:34.737 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:34.737 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:34.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:34.741 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:34.742 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:34.742 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:34.742 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:34.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:34.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:34.812 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:34.812 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:26:34.813 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:34.813 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:34.813 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:34.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:34.824 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:34.825 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:34.825 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:34.825 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:34.825 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:34.826 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:34.826 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:34.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:34.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:34.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:34.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:34.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2021-07-26T06:26:34.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:26:34.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:34.932 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:34.932 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:34.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:34.948 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:34.948 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:34.949 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:34.949 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:34.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:34.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:35.021 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:35.021 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:26:35.021 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:35.022 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:35.022 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:35.067 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:35.067 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:35.067 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:35.068 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:35.068 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:35.068 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:35.068 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:35.068 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:35.069 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:35.069 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:35.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:35.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:35.138 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2021-07-26T06:26:35.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:26:35.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:35.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:35.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:35.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:35.150 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:35.151 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:35.151 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:35.151 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:35.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:35.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:35.231 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:35.231 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:35.231 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:35.232 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:35.232 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:35.245 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:35.245 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:35.245 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:35.246 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:35.246 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:35.246 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:35.246 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:35.246 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:35.247 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:35.247 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:35.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:35.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:35.343 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2021-07-26T06:26:35.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:26:35.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:35.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:35.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:35.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:35.346 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:35.346 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:35.346 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:35.347 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:35.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:35.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:35.443 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:35.443 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:35.444 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:26:35.444 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:35.444 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:35.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:35.457 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:35.458 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:35.458 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:35.458 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:35.459 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:35.459 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:35.459 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:35.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:35.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:35.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:26:35.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:26:35.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 1 2021-07-26T06:26:35.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:26:35.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:35.567 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:35.567 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:35.567 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:35.567 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:35.568 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:35.568 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:35.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:35.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:35.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:35.662 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:35.662 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:35.664 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:26:35.664 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:35.665 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:35.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:35.666 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:35.667 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:35.667 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:35.667 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:35.667 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:35.667 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:35.668 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:35.668 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:35.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:35.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:35.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:35.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 4 2021-07-26T06:26:35.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:26:35.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:35.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:35.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:35.778 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:35.778 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:35.779 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:35.779 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:35.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:35.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:35.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:35.863 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:35.863 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:35.863 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:26:35.864 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:35.864 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:35.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:35.874 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:35.874 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:35.874 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:35.875 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:35.875 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:35.875 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:35.885 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:35.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:35.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:35.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:26:35.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:26:35.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 15 + 0 2021-07-26T06:26:35.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:26:35.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2021-07-26T06:26:35.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:26:35.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 504 + 15 2021-07-26T06:26:36.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=519 2021-07-26T06:26:36.019 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:26:36.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:26:39.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:26:39.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:26:39.995 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:26:39.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:26:39.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:39.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:39.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:39.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:39.998 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:39.999 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:39.999 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:40.000 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:40.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:40.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:40.101 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:40.102 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:40.102 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:40.102 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:40.103 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:40.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:40.105 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:40.105 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:40.105 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:40.105 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:40.106 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:40.106 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:40.106 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:40.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:40.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:40.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:40.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:40.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2021-07-26T06:26:40.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:26:40.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:40.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:40.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:40.206 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:40.206 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:40.206 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:40.211 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:40.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:40.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:40.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:40.300 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:40.300 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:26:40.300 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:40.301 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:40.301 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:40.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:40.312 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:40.312 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:40.312 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:40.312 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:40.313 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:40.315 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:40.316 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:40.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:40.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:40.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:40.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:40.410 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2021-07-26T06:26:40.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:26:40.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:40.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:40.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:40.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:40.413 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:40.414 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:40.414 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:40.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:40.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:40.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:40.518 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:40.519 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:40.519 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:40.519 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:40.520 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:40.521 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:40.522 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:40.522 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:40.522 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:40.522 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:40.523 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:40.523 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:40.523 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:40.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:40.524 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:40.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:26:40.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:26:40.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2021-07-26T06:26:40.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:26:40.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:40.630 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:40.630 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:40.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:40.631 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:40.631 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:40.631 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:40.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:40.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:40.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:40.716 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:40.716 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:40.717 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:26:40.718 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:40.718 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:40.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:40.727 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:40.734 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:40.736 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:40.737 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:40.737 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:40.738 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:40.738 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:40.738 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:40.738 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:40.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:26:40.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:26:40.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 1 2021-07-26T06:26:40.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:26:40.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:40.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:40.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:40.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:40.875 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:40.875 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:40.875 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:40.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:40.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:40.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:40.926 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:40.927 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:40.927 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:26:40.927 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:40.927 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:40.938 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:40.938 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:40.938 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:40.939 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:40.939 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:40.939 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:40.939 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:40.948 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:40.949 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:40.949 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:41.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:41.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:41.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2021-07-26T06:26:41.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:26:41.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:41.042 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:41.042 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:41.043 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:41.043 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:41.043 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:41.043 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:41.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:41.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:41.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:41.136 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:41.137 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:41.137 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:41.137 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:41.137 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:41.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:41.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:41.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:41.149 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:41.149 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:41.150 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:41.150 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:41.151 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:41.151 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:41.151 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:41.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:26:41.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:26:41.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 1 2021-07-26T06:26:41.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:26:41.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2021-07-26T06:26:41.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:26:41.250 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 519 + 15 2021-07-26T06:26:41.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=534 2021-07-26T06:26:41.251 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:26:41.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:26:45.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:26:45.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:26:45.255 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:26:45.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:26:45.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:45.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:45.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:45.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:45.265 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:45.273 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:45.274 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:45.274 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:45.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:45.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:45.362 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:45.362 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:45.363 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:45.363 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:45.363 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:45.373 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:45.374 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:45.374 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:45.374 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:45.374 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:45.375 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:45.375 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:45.375 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:45.375 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:45.376 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:45.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:45.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:45.481 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2021-07-26T06:26:45.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:26:45.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:45.481 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:45.481 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:45.482 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:45.482 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:45.484 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:45.484 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:45.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:45.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:45.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:45.579 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:45.580 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:26:45.580 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:45.580 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:45.581 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:45.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:45.588 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:45.596 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:45.597 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:45.597 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:45.598 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:45.599 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:45.599 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:45.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:45.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:45.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:45.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:45.688 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 3 2021-07-26T06:26:45.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2021-07-26T06:26:45.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:45.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:45.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:45.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:45.691 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:45.691 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:45.691 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:45.691 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:45.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:45.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:45.783 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:45.783 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:45.784 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:45.784 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:45.784 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:45.796 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:45.796 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:45.797 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:45.797 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:45.797 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:45.797 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:45.798 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:45.798 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:45.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:45.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:45.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:26:45.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:26:45.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 1 2021-07-26T06:26:45.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:26:45.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:45.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:45.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:45.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:45.915 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:45.915 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:45.916 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:45.916 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:45.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:45.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:46.001 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:46.002 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:46.002 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:26:46.002 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:46.003 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:46.004 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:46.004 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:46.004 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:46.005 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:46.005 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:46.005 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:46.005 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:46.006 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:46.006 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:46.006 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:46.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:26:46.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:26:46.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 1 2021-07-26T06:26:46.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:26:46.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:46.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:46.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:46.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:46.110 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:46.110 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:46.110 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:46.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:46.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:46.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:46.202 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:46.202 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:46.202 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:46.202 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:46.203 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:46.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:46.212 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:46.213 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:46.213 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:46.213 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:46.213 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:46.214 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:46.214 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:46.214 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:46.214 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:46.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:46.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:46.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 3 2021-07-26T06:26:46.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:26:46.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:46.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:46.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:46.319 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:46.319 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:46.319 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:46.319 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:46.319 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:46.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:46.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:46.411 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:46.412 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:46.412 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:46.412 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:46.412 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:46.422 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:46.422 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:46.422 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:46.426 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:46.426 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:46.426 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:46.427 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:46.427 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:46.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:46.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:46.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:26:46.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:26:46.535 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 1 2021-07-26T06:26:46.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:26:46.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2021-07-26T06:26:46.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:26:46.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 534 + 12 2021-07-26T06:26:46.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=546 2021-07-26T06:26:46.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:26:46.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:26:50.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:26:50.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:26:50.531 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:26:50.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:26:50.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:50.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:50.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:50.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:50.535 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:50.535 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:50.535 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:50.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:50.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:50.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:50.639 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:50.639 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:50.640 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:50.640 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:50.640 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:50.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:50.651 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:50.651 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:50.651 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:50.652 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:50.652 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:50.652 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:50.652 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:50.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:50.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:50.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:50.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:50.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2021-07-26T06:26:50.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:26:50.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:50.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:50.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:50.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:50.762 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:50.762 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:50.763 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:50.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:50.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:50.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:50.849 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:50.849 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:26:50.850 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:50.852 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:50.853 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:50.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:50.860 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:50.861 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:50.861 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:50.861 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:50.861 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:50.861 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:50.862 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:50.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:50.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:50.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:50.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:50.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2021-07-26T06:26:50.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:26:50.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:50.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:50.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:50.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:50.971 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:50.971 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:50.971 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:50.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:50.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:50.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:51.059 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:51.059 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:51.061 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:51.062 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:51.062 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:51.076 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:51.076 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:51.078 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:51.078 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:51.079 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:51.079 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:51.079 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:51.079 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:51.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:51.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:51.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:26:51.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:26:51.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2021-07-26T06:26:51.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:26:51.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:51.178 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:51.178 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:51.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:51.179 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:51.179 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:51.179 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:51.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:51.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:51.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:51.270 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:51.282 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:51.283 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:26:51.283 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:51.283 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:51.286 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:51.286 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:51.287 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:51.287 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:51.288 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:51.288 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:51.289 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:51.290 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:51.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:51.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:51.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:26:51.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:26:51.384 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 1 2021-07-26T06:26:51.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:26:51.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:51.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:51.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:51.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:51.391 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:51.391 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:51.391 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:51.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:51.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:51.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:51.487 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:51.487 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:51.487 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:26:51.488 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:51.488 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:51.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:51.497 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:51.498 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:51.498 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:51.498 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:51.498 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:51.499 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:51.499 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:51.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:51.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:51.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:51.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:51.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2021-07-26T06:26:51.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:26:51.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:51.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:51.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:51.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:51.611 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:51.611 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:51.611 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:51.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:51.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:51.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:51.691 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:51.692 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:51.692 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:26:51.692 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:51.693 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:51.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:51.702 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:51.702 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:51.703 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:51.703 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:51.704 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:51.705 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:51.705 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:51.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:51.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:51.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:26:51.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:26:51.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 1 2021-07-26T06:26:51.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:26:51.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2021-07-26T06:26:51.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:26:51.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 546 + 15 2021-07-26T06:26:51.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=561 2021-07-26T06:26:51.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:26:51.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:26:55.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:26:55.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:26:55.812 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:26:55.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:26:55.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:55.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:55.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:55.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:55.831 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:55.831 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:55.831 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:55.831 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:55.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:55.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:55.918 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:55.919 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:26:55.919 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:55.919 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:55.919 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:55.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:55.928 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:26:55.928 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:26:55.929 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:26:55.929 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:55.929 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:55.929 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:55.938 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:26:55.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:55.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:26:56.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:26:56.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:26:56.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2021-07-26T06:26:56.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:26:56.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:56.052 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:56.053 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:56.053 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:56.055 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:56.055 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:56.056 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:56.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:56.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:56.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:56.131 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:56.131 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:26:56.132 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:56.132 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:56.133 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:56.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:56.142 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:26:56.142 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:26:56.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:26:56.169 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:56.169 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:56.169 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:56.170 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:26:56.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:56.171 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:26:56.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:56.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:56.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2021-07-26T06:26:56.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:26:56.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:56.249 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:56.249 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:56.250 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:56.250 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:56.250 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:56.251 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:56.251 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:56.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:56.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:56.339 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:56.339 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:56.340 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:56.340 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:56.340 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:56.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:56.352 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:26:56.352 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:26:56.371 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:26:56.372 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:56.389 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:56.391 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:56.408 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:26:56.408 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:56.408 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:26:56.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:26:56.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:26:56.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2021-07-26T06:26:56.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:26:56.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:56.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:56.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:56.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:56.470 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:56.470 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:56.471 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:56.471 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:56.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:56.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:56.552 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:56.553 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:56.553 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:26:56.553 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:56.553 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:56.567 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:56.568 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:26:56.568 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:26:56.568 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:26:56.569 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:56.569 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:56.569 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:56.569 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:26:56.569 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:56.570 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:26:56.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:26:56.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:26:56.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 0 2021-07-26T06:26:56.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2021-07-26T06:26:56.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:56.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:56.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:56.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:56.664 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:56.665 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:56.665 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:56.665 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:56.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:56.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:56.764 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:56.765 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:26:56.766 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:56.766 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:56.766 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:56.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:56.768 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:26:56.768 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:26:56.768 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:26:56.769 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:56.769 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:56.769 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:56.769 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:26:56.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:56.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:26:56.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:26:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:26:56.871 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 4 2021-07-26T06:26:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2021-07-26T06:26:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:26:56.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:56.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:56.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:56.875 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:56.875 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:56.875 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:56.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:56.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:56.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:56.969 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:26:56.969 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:26:56.970 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:26:56.970 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:26:56.970 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:26:56.980 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:26:56.981 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:26:56.981 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:26:56.981 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:26:56.981 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:26:56.982 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:56.983 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:26:56.984 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:26:56.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:26:56.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:26:57.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:26:57.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:26:57.088 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 2 2021-07-26T06:26:57.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:26:57.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2021-07-26T06:26:57.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:26:57.091 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 561 + 15 2021-07-26T06:26:57.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=576 2021-07-26T06:26:57.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:26:57.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:27:01.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:27:01.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:27:01.091 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:27:01.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:27:01.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:01.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:01.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:01.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:01.103 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:01.104 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:01.104 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:01.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:01.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:01.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:01.199 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:01.199 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:01.200 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:27:01.200 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:01.200 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:01.210 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:01.210 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:01.210 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:01.210 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:01.211 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:01.211 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:01.211 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:01.211 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:01.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:01.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:01.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:01.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:01.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2021-07-26T06:27:01.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2021-07-26T06:27:01.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:01.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:01.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:01.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:01.318 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:01.318 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:01.319 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:01.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:01.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:01.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:01.411 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:01.412 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:27:01.412 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:01.412 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:01.412 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:01.423 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:01.436 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:01.437 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:01.437 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:01.438 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:01.438 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:01.438 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:01.438 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:01.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:01.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:01.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:01.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:01.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 2 2021-07-26T06:27:01.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:27:01.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:01.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:01.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:01.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:01.530 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:01.530 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:01.530 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:01.531 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:01.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:01.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:01.622 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:01.622 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:01.623 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:27:01.623 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:01.623 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:01.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:01.633 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:01.634 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:01.634 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:01.634 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:01.634 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:01.635 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:01.635 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:01.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:01.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:01.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:27:01.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:27:01.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2021-07-26T06:27:01.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:27:01.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:01.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:01.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:01.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:01.761 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:01.761 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:01.761 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:01.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:01.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:01.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:01.832 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:01.832 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:01.833 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:01.833 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:01.833 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:01.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:01.845 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:01.845 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:01.845 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:01.845 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:01.847 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:01.847 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:01.847 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:01.847 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:01.848 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:01.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:01.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:01.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 0 2021-07-26T06:27:01.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:27:01.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:01.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:01.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:01.949 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:01.949 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:01.950 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:01.950 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:01.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:01.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:01.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:02.044 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:02.044 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 4, 2021-07-26T06:27:02.044 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:02.045 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:02.045 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:02.053 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:02.054 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:02.054 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:02.054 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:02.056 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:02.056 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:02.056 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:02.057 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:02.057 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:02.057 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:02.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:02.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:02.159 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 4 2021-07-26T06:27:02.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:27:02.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:02.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:02.161 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:02.161 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:02.162 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:02.162 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:02.162 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:02.162 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:02.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:02.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:02.253 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:02.254 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:02.254 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:27:02.254 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:02.255 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:02.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:02.264 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:02.264 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:02.264 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:02.264 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:02.265 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:02.265 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:02.265 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:02.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:02.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:02.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:02.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:02.368 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 4 2021-07-26T06:27:02.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:27:02.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2021-07-26T06:27:02.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:27:02.369 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 576 + 15 2021-07-26T06:27:02.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=591 2021-07-26T06:27:02.369 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:27:02.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:27:06.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:27:06.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:27:06.372 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:27:06.373 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:27:06.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:06.376 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:06.376 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:06.377 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:06.377 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:06.377 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:06.377 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:06.378 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:06.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:06.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:06.480 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:06.481 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:06.481 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:27:06.481 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:06.481 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:06.492 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:06.492 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:06.493 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:06.493 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:06.493 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:06.493 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:06.493 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:06.494 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:06.494 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:06.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:06.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:06.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:06.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2021-07-26T06:27:06.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2021-07-26T06:27:06.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:06.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:06.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:06.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:06.599 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:06.599 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:06.600 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:06.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:06.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:06.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:06.691 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:06.691 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:27:06.692 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:06.692 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:06.692 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:06.703 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:06.703 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:06.704 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:06.704 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:06.705 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:06.705 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:06.706 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:06.706 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:06.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:06.707 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:06.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:06.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:06.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 2 2021-07-26T06:27:06.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:27:06.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:06.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:06.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:06.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:06.809 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:06.809 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:06.809 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:06.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:06.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:06.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:06.902 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:06.902 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:06.902 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:27:06.902 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:06.903 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:06.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:06.912 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:06.913 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:06.913 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:06.913 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:06.913 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:06.913 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:06.914 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:06.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:06.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:07.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:07.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:07.015 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2021-07-26T06:27:07.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:27:07.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:07.017 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:07.018 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:07.018 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:07.018 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:07.018 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:07.019 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:07.019 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:07.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:07.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:07.111 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:07.112 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:07.112 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:07.112 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:07.112 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:07.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:07.122 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:07.122 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:07.123 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:07.123 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:07.123 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:07.123 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:07.124 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:07.124 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:07.124 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:07.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:07.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:07.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 0 2021-07-26T06:27:07.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:27:07.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:07.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:07.233 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:07.233 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:07.234 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:07.234 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:07.234 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:07.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:07.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:07.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:07.321 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:07.321 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 4, 2021-07-26T06:27:07.321 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:07.322 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:07.323 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:07.332 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:07.332 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:07.333 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:07.333 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:07.333 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:07.334 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:07.334 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:07.335 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:07.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:07.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:07.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:07.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:07.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 4 2021-07-26T06:27:07.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:27:07.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:07.438 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:07.438 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:07.438 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:07.439 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:07.439 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:07.439 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:07.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:07.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:07.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:07.532 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:07.533 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:07.533 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:27:07.533 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:07.533 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:07.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:07.543 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:07.543 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:07.543 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:07.543 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:07.544 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:07.544 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:07.544 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:07.544 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:07.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:07.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:27:07.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:27:07.645 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 3 2021-07-26T06:27:07.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:27:07.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2021-07-26T06:27:07.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:27:07.647 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 591 + 15 2021-07-26T06:27:07.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=606 2021-07-26T06:27:07.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:27:07.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:27:11.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:27:11.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:27:11.652 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:27:11.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:27:11.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:11.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:11.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:11.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:11.657 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:11.657 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:11.657 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:11.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:11.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:11.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:11.758 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:11.762 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:11.762 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:27:11.763 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:11.763 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:11.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:11.771 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:11.771 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:11.771 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:11.771 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:11.772 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:11.772 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:11.772 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:11.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:11.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:11.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:11.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:11.873 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2021-07-26T06:27:11.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2021-07-26T06:27:11.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:11.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:11.880 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:11.880 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:11.880 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:11.881 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:11.881 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:11.882 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:11.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:11.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:11.970 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:11.970 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:27:11.970 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:11.971 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:11.971 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:11.980 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:11.981 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:11.981 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:11.981 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:11.982 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:11.982 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:11.988 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:11.989 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:11.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:11.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:12.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:12.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:12.083 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 2 2021-07-26T06:27:12.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:27:12.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:12.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:12.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:12.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:12.086 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:12.086 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:12.087 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:12.088 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:12.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:12.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:12.179 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:12.180 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:12.180 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:27:12.180 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:12.181 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:12.191 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:12.191 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:12.191 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:12.192 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:12.192 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:12.192 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:12.192 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:12.193 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:12.193 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:12.193 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:12.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:12.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:12.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2021-07-26T06:27:12.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:27:12.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:12.296 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:12.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:12.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:12.298 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:12.298 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:12.298 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:12.298 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:12.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:12.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:12.396 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:12.397 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:12.397 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:12.397 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:12.397 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:12.402 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:12.402 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:12.402 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:12.403 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:12.403 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:12.403 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:12.403 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:12.403 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:12.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:12.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:12.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:12.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:12.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 0 2021-07-26T06:27:12.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:27:12.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:12.507 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:12.507 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:12.507 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:12.507 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:12.508 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:12.508 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:12.508 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:12.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:12.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:12.601 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:12.601 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 4, 2021-07-26T06:27:12.601 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:12.602 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:12.602 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:12.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:12.612 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:12.613 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:12.613 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:12.613 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:12.613 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:12.614 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:12.614 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:12.614 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:12.614 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:12.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:12.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:12.715 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 4 2021-07-26T06:27:12.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:27:12.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:12.717 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:12.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:12.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:12.718 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:12.719 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:12.719 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:12.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:12.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:12.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:12.810 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:12.810 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:12.810 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:27:12.810 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:12.811 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:12.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:12.821 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:12.821 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:12.821 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:12.821 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:12.822 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:12.822 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:12.822 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:12.822 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:12.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:12.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:27:12.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:27:12.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 3 2021-07-26T06:27:12.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:27:12.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2021-07-26T06:27:12.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:27:12.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 606 + 15 2021-07-26T06:27:12.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=621 2021-07-26T06:27:12.926 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:27:12.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:27:16.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:27:16.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:27:16.931 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:27:16.932 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:27:16.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:16.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:16.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:16.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:16.936 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:16.937 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:16.937 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:16.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:16.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:16.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:17.040 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:17.040 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:17.041 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:27:17.041 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:17.041 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:17.049 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:17.050 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:17.050 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:17.051 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:17.051 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:17.051 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:17.051 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:17.052 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:17.052 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:17.052 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:17.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:27:17.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:27:17.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2021-07-26T06:27:17.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:27:17.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:17.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:17.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:17.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:17.156 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:17.156 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:17.157 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:17.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:17.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:17.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:17.250 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:17.251 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:27:17.251 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:17.251 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:17.251 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:17.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:17.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:17.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:17.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:17.262 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:17.264 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:17.264 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:17.265 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:17.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:17.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:17.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:17.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:17.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2021-07-26T06:27:17.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2021-07-26T06:27:17.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:17.367 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:17.367 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:17.367 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:17.368 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:17.369 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:17.369 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:17.369 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:17.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:17.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:17.462 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:17.462 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:17.462 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:27:17.463 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:17.463 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:17.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:17.472 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:17.473 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:17.473 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:17.473 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:17.473 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:17.474 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:17.474 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:17.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:17.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:17.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:17.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:17.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 4 2021-07-26T06:27:17.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:27:17.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:17.576 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:17.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:17.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:17.577 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:17.577 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:17.577 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:17.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:17.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:17.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:17.670 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:17.670 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:17.671 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:17.672 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:17.672 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:17.682 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:17.682 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:17.683 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:17.683 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:17.683 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:17.683 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:17.684 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:17.684 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:17.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:17.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:17.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:27:17.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:27:17.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 1 2021-07-26T06:27:17.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:27:17.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:17.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:17.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:17.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:17.785 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:17.786 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:17.786 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:17.786 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:17.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:17.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:17.878 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:17.879 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:27:17.879 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:17.879 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:17.880 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:17.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:17.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:17.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:17.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:17.890 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:17.890 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:17.890 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:17.890 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:17.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:17.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:17.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:27:17.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:27:17.992 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 3 2021-07-26T06:27:17.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:27:17.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:17.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:17.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:17.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:17.995 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:17.995 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:17.995 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:17.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:17.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:17.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:18.088 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:18.089 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:18.089 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:27:18.089 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:18.090 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:18.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:18.099 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:18.100 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:18.100 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:18.100 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:18.100 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:18.101 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:18.101 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:18.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:18.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:18.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:18.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:18.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 15 + 2 2021-07-26T06:27:18.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2021-07-26T06:27:18.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 17 = 0 ']' 2021-07-26T06:27:18.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:27:18.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 621 + 17 2021-07-26T06:27:18.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=638 2021-07-26T06:27:18.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:27:18.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:27:22.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:27:22.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:27:22.209 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:27:22.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:27:22.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:22.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:22.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:22.214 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:22.214 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:22.214 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:22.215 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:22.215 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:22.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:22.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:22.316 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:22.316 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:22.317 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:27:22.317 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:22.317 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:22.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:22.327 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:22.327 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:22.328 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:22.329 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:22.329 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:22.329 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:22.330 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:22.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:22.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:22.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:22.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:22.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:27:22.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:27:22.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:22.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:22.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:22.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:22.435 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:22.435 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:22.436 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:22.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:22.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:22.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:22.530 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:22.531 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:27:22.531 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:22.531 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:22.532 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:22.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:22.540 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:22.541 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:22.541 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:22.541 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:22.541 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:22.543 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:22.544 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:22.544 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:22.544 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:22.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:22.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:22.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2021-07-26T06:27:22.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:27:22.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:22.647 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:22.647 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:22.647 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:22.647 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:22.648 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:22.648 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:22.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:22.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:22.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:22.738 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:22.739 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:22.739 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:27:22.739 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:22.739 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:22.748 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:22.749 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:22.749 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:22.749 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:22.749 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:22.750 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:22.750 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:22.750 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:22.750 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:22.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:22.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:22.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:22.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 4 2021-07-26T06:27:22.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:27:22.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:22.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:22.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:22.856 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:22.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:22.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:22.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:22.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:22.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:22.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:22.948 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:22.948 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:22.948 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:22.948 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:22.949 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:22.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:22.959 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:22.959 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:22.959 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:22.959 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:22.961 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:22.962 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:22.962 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:22.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:22.963 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:23.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:27:23.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:27:23.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 1 2021-07-26T06:27:23.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2021-07-26T06:27:23.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:23.065 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:23.065 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:23.066 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:23.066 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:23.066 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:23.066 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:23.067 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:23.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:23.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:23.158 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:23.158 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:27:23.158 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:23.159 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:23.159 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:23.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:23.168 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:23.169 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:23.169 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:23.169 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:23.169 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:23.170 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:23.170 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:23.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:23.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:23.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:27:23.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:27:23.272 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 3 2021-07-26T06:27:23.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=16 2021-07-26T06:27:23.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:23.275 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:23.275 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:23.275 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:23.275 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:23.276 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:23.276 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:23.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:23.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:23.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:23.368 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:23.368 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:23.369 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:27:23.369 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:23.369 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:23.379 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:23.379 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:23.379 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:23.379 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:23.380 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:23.380 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:23.380 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:23.380 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:23.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:23.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:23.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:23.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:23.483 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 16 + 2 2021-07-26T06:27:23.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2021-07-26T06:27:23.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2021-07-26T06:27:23.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:27:23.485 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 638 + 18 2021-07-26T06:27:23.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=656 2021-07-26T06:27:23.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:27:23.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:27:27.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:27:27.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:27:27.490 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:27:27.490 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:27:27.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:27.493 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:27.494 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:27.494 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:27.494 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:27.495 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:27.495 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:27.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:27.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:27.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:27.597 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:27.597 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:27.598 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:27:27.598 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:27.598 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:27.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:27.608 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:27.609 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:27.609 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:27.609 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:27.609 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:27.610 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:27.610 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:27.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:27.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:27.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:27.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:27.711 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:27:27.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:27:27.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:27.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:27.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:27.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:27.714 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:27.715 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:27.715 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:27.715 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:27.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:27.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:27.809 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:27.809 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:27:27.810 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:27.810 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:27.810 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:27.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:27.820 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:27.820 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:27.820 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:27.821 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:27.821 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:27.821 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:27.821 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:27.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:27.822 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:27.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:27.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:27.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2021-07-26T06:27:27.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:27:27.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:27.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:27.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:27.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:27.924 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:27.924 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:27.925 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:27.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:27.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:27.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:28.017 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:28.017 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:28.017 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 3, 2021-07-26T06:27:28.018 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:28.018 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:28.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:28.028 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:28.028 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:28.028 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:28.028 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:28.029 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:28.029 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:28.029 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:28.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:28.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:28.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2021-07-26T06:27:28.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2021-07-26T06:27:28.131 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 3 2021-07-26T06:27:28.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:27:28.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:28.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:28.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:28.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:28.134 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:28.134 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:28.134 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:28.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:28.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:28.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:28.227 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:28.227 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:28.227 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:28.228 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:28.228 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:28.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:28.237 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:28.237 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:28.238 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:28.238 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:28.238 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:28.238 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:28.239 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:28.239 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:28.239 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:28.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:27:28.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:27:28.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 1 2021-07-26T06:27:28.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:27:28.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:28.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:28.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:28.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:28.344 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:28.345 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:28.345 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:28.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:28.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:28.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:28.438 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:28.438 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:27:28.438 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:28.439 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:28.439 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:28.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:28.448 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:28.449 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:28.449 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:28.449 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:28.449 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:28.450 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:28.450 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:28.450 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:28.450 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:28.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:28.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:28.552 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 2 2021-07-26T06:27:28.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2021-07-26T06:27:28.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:28.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:28.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:28.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:28.556 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:28.556 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:28.556 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:28.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:28.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:28.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:28.648 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:28.648 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:28.649 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:28.650 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:28.650 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:28.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:28.659 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:28.659 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:28.659 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:28.659 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:28.660 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:28.660 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:28.660 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:28.660 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:28.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:28.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:27:28.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:27:28.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 1 2021-07-26T06:27:28.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2021-07-26T06:27:28.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2021-07-26T06:27:28.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:27:28.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 656 + 15 2021-07-26T06:27:28.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=671 2021-07-26T06:27:28.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:27:28.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:27:32.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:27:32.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:27:32.766 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:27:32.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:27:32.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:32.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:32.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:32.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:32.771 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:32.771 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:32.772 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:32.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:32.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:32.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:32.874 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:32.874 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:32.875 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:27:32.875 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:32.875 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:32.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:32.885 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:32.885 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:32.886 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:32.886 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:32.886 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:32.886 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:32.887 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:32.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:32.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:32.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:32.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:32.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:27:32.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:27:32.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:32.992 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:32.992 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:32.992 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:32.993 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:32.993 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:32.993 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:32.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:32.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:32.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:33.086 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:33.087 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:27:33.087 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:33.087 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:33.088 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:33.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:33.096 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:33.097 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:33.097 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:33.097 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:33.097 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:33.098 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:33.098 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:33.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:33.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:33.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:33.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:33.199 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2021-07-26T06:27:33.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:27:33.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:33.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:33.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:33.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:33.203 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:33.203 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:33.203 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:33.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:33.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:33.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:33.295 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:33.295 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:33.295 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:27:33.296 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:33.296 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:33.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:33.305 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:33.305 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:33.306 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:33.306 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:33.306 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:33.306 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:33.307 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:33.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:33.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:33.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:33.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:33.410 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 2 2021-07-26T06:27:33.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:27:33.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:33.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:33.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:33.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:33.413 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:33.413 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:33.413 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:33.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:33.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:33.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:33.506 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:33.506 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:33.506 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:33.507 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:33.507 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:33.516 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:33.517 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:33.517 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:33.518 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:33.519 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:33.519 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:33.519 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:33.520 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:33.520 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:33.520 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:33.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:27:33.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:27:33.620 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 1 2021-07-26T06:27:33.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:27:33.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:33.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:33.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:33.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:33.623 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:33.623 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:33.623 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:33.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:33.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:33.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:33.714 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:33.715 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:27:33.715 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:33.715 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:33.715 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:33.724 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:33.725 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:33.725 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:33.725 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:33.726 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:33.726 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:33.726 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:33.726 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:33.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:33.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:33.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:27:33.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:27:33.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 1 2021-07-26T06:27:33.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:27:33.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:33.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:33.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:33.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:33.830 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:33.830 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:33.830 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:33.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:33.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:33.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:33.923 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:33.923 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:33.923 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:33.924 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:33.924 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:33.933 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:33.933 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:33.933 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:33.934 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:33.934 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:33.934 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:33.934 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:33.935 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:33.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:33.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:34.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:34.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:34.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 0 2021-07-26T06:27:34.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:27:34.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2021-07-26T06:27:34.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:27:34.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 671 + 12 2021-07-26T06:27:34.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=683 2021-07-26T06:27:34.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:27:34.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:27:38.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:27:38.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:27:38.043 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:27:38.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:27:38.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:38.047 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:38.047 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:38.047 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:38.048 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:38.048 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:38.048 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:38.048 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:38.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:38.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:38.150 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:38.151 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:38.151 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 4, 2021-07-26T06:27:38.151 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:38.151 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:38.161 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:38.162 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:38.162 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:38.162 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:38.162 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:38.163 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:38.163 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:38.163 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:38.164 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:38.164 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:38.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:38.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:38.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2021-07-26T06:27:38.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:27:38.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:38.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:38.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:38.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:38.267 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:38.268 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:38.268 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:38.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:38.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:38.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:38.361 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:38.361 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 3, 2021-07-26T06:27:38.361 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:38.362 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:38.362 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:38.371 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:38.372 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:38.372 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:38.372 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:38.372 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:38.372 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:38.373 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:38.373 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:38.373 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:38.373 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:38.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2021-07-26T06:27:38.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2021-07-26T06:27:38.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2021-07-26T06:27:38.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2021-07-26T06:27:38.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:38.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:38.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:38.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:38.477 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:38.478 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:38.478 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:38.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:38.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:38.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:38.571 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:38.571 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:38.572 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:27:38.572 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:38.572 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:38.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:38.582 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:38.582 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:38.582 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:38.583 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:38.583 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:38.583 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:38.583 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:38.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:38.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:38.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:38.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:38.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 2 2021-07-26T06:27:38.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2021-07-26T06:27:38.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:38.686 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:38.687 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:38.687 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:38.687 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:38.687 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:38.688 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:38.688 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:38.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:38.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:38.779 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:38.780 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:38.780 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:38.780 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:38.780 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:38.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:38.790 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:38.790 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:38.790 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:38.790 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:38.791 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:38.791 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:38.791 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:38.791 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:38.791 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:38.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:27:38.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:27:38.893 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 1 2021-07-26T06:27:38.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2021-07-26T06:27:38.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:38.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:38.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:38.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:38.896 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:38.896 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:38.896 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:38.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:38.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:38.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:38.989 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:38.989 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:27:38.989 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:38.990 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:38.990 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:38.999 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:38.999 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:39.000 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:39.000 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:39.000 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:39.000 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:39.001 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:39.001 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:39.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:39.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:39.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:27:39.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:27:39.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 1 2021-07-26T06:27:39.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:27:39.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:39.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:39.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:39.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:39.107 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:39.107 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:39.108 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:39.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:39.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:39.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:39.198 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:39.199 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:39.199 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:39.199 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:39.200 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:39.209 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:39.209 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:39.210 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:39.210 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:39.210 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:39.210 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:39.211 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:39.211 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:39.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:39.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:39.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:39.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:39.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 0 2021-07-26T06:27:39.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2021-07-26T06:27:39.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2021-07-26T06:27:39.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:27:39.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 683 + 12 2021-07-26T06:27:39.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=695 2021-07-26T06:27:39.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:27:39.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:27:43.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:27:43.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:27:43.318 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:27:43.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:27:43.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:43.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:43.323 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:43.323 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:43.323 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:43.323 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:43.324 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:43.324 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:43.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:43.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:43.427 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:43.427 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:43.427 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:27:43.428 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:43.428 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:43.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:43.439 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:43.439 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:43.440 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:43.440 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:43.440 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:43.440 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:43.440 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:43.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:43.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:43.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:43.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:43.543 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2021-07-26T06:27:43.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2021-07-26T06:27:43.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:43.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:43.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:43.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:43.545 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:43.546 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:43.546 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:43.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:43.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:43.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:43.637 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:43.637 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 2, 2021-07-26T06:27:43.638 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:43.638 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:43.638 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:43.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:43.649 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:43.649 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:43.649 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:43.650 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:43.650 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:43.650 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:43.650 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:43.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:43.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:43.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:43.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:43.748 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 2 2021-07-26T06:27:43.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2021-07-26T06:27:43.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:43.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:43.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:43.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:43.752 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:43.752 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:43.753 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:43.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:43.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:43.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:43.844 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:43.845 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:43.845 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 2, 2021-07-26T06:27:43.845 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:43.846 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:43.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:43.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:43.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:43.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:43.856 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:43.857 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:43.857 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:43.857 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:43.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:43.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:43.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2021-07-26T06:27:43.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2021-07-26T06:27:43.959 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2021-07-26T06:27:43.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2021-07-26T06:27:43.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:43.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:43.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:43.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:43.962 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:43.963 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:43.963 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:43.963 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:43.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:43.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:44.055 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:44.055 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:44.055 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:44.056 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:44.056 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:44.065 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:44.066 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:44.067 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:44.067 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:44.067 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:44.068 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:44.068 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:44.068 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:44.068 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:44.069 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:44.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:44.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:44.169 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 0 2021-07-26T06:27:44.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2021-07-26T06:27:44.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:44.171 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:44.172 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:44.172 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:44.172 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:44.172 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:44.173 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:44.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:44.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:44.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:44.265 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:44.266 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:44.266 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:44.266 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:44.267 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:44.275 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:44.276 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:44.276 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:44.276 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:44.276 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:44.277 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:44.277 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:44.277 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:44.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:44.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:44.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:44.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:44.379 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 0 2021-07-26T06:27:44.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2021-07-26T06:27:44.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:44.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:44.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:44.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:44.382 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:44.382 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:44.382 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:44.382 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:44.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:44.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:44.474 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:44.474 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:44.474 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:44.475 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:44.475 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:44.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:44.485 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:44.485 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:44.485 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:44.485 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:44.486 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:44.486 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:44.486 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:44.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:44.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:44.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:44.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:44.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 0 2021-07-26T06:27:44.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2021-07-26T06:27:44.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 6 = 0 ']' 2021-07-26T06:27:44.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:27:44.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 695 + 6 2021-07-26T06:27:44.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=701 2021-07-26T06:27:44.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:27:44.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:27:48.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:27:48.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:27:48.594 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:27:48.595 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:27:48.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:48.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:48.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:48.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:48.600 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:48.600 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:48.600 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:48.601 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:48.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:48.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:48.702 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:48.702 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:48.702 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:48.703 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:48.703 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:48.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:48.714 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:48.714 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:48.715 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:48.715 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:48.715 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:48.715 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:48.716 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:48.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:48.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:48.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:27:48.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:27:48.817 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2021-07-26T06:27:48.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2021-07-26T06:27:48.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:48.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:48.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:48.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:48.820 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:48.820 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:48.820 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:48.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:48.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:48.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:48.915 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:48.916 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 1, 2021-07-26T06:27:48.916 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:48.916 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:48.916 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:48.926 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:48.926 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:48.926 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:48.926 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:48.927 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:48.927 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:48.927 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:48.928 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:48.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:48.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:49.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:27:49.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:27:49.030 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 1 2021-07-26T06:27:49.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2021-07-26T06:27:49.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:49.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:49.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:49.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:49.033 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:49.033 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:49.033 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:49.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:49.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:49.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:49.126 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:49.127 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:49.127 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 1, 2021-07-26T06:27:49.127 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:49.127 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:49.137 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:49.137 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:49.137 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:49.137 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:49.138 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:49.138 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:49.139 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:49.139 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:49.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:49.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:49.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2021-07-26T06:27:49.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2021-07-26T06:27:49.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 1 2021-07-26T06:27:49.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:27:49.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:49.243 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:49.243 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:49.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:49.244 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:49.245 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:49.246 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:49.246 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:49.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:49.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:49.338 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:49.338 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:49.338 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:49.338 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:49.339 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:49.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:49.349 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:49.349 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:49.349 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:49.350 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:49.350 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:49.352 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:49.353 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:49.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:49.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:49.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:49.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:49.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 0 2021-07-26T06:27:49.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:27:49.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:49.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:49.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:49.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:49.455 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:49.456 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:49.456 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:49.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:49.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:49.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:49.549 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:49.549 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:49.549 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:49.550 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:49.550 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:49.563 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:49.564 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:49.564 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:49.564 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:49.564 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:49.565 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:49.565 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:49.565 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:49.565 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:49.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:49.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:49.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:49.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 0 2021-07-26T06:27:49.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:27:49.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:49.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:49.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:49.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:49.664 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:49.665 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:49.665 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:49.665 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:49.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:49.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:49.757 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:49.757 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:49.758 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:49.758 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:49.758 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:49.767 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:49.768 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:49.768 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:49.768 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:49.768 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:49.769 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:49.769 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:49.769 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:49.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:49.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:49.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:49.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:49.870 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 0 2021-07-26T06:27:49.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2021-07-26T06:27:49.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 3 = 0 ']' 2021-07-26T06:27:49.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2021-07-26T06:27:49.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 701 + 3 2021-07-26T06:27:49.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=704 2021-07-26T06:27:49.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:27:49.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:27:53.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:27:53.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:27:53.880 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:27:53.880 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:27:53.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:53.881 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:53.882 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:53.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:53.883 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:53.883 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:53.883 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:53.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:53.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:53.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:53.985 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:53.986 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:53.986 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:53.988 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:53.989 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:53.999 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:53.999 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:53.999 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:54.000 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:54.000 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:54.001 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:54.002 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:54.002 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:54.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:54.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:54.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:54.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:54.100 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:27:54.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:27:54.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:54.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:54.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:54.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:54.104 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:54.104 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:54.105 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:54.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:54.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:54.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:54.197 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:54.197 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:54.198 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:54.198 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:54.198 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:54.207 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:54.208 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:54.208 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:54.208 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:54.208 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:54.208 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:54.209 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:54.209 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:54.209 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:54.209 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:54.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:54.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:54.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:27:54.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:27:54.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:54.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:54.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:54.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:54.313 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:54.313 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:54.313 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:54.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:54.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:54.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:54.407 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:54.407 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:54.408 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:54.408 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:54.408 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:54.419 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:54.419 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:54.419 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:54.419 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:54.420 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:54.420 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:54.420 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:54.420 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:54.421 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:54.421 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:54.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:54.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:54.521 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:27:54.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:27:54.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:54.524 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:54.524 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:54.524 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:54.525 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:54.525 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:54.525 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:54.525 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:54.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:54.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:54.618 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:54.618 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:54.619 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:54.619 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:54.619 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:54.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:54.629 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:54.629 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:54.630 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:54.630 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:54.630 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:54.630 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:54.631 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:54.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:54.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:54.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:54.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:54.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:27:54.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:27:54.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:54.733 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:54.733 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:54.733 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:54.734 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:54.734 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:54.734 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:54.734 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:54.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:54.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:54.826 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:54.827 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:54.827 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:54.827 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:54.828 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:54.837 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:54.837 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:27:54.837 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:27:54.838 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:27:54.838 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:54.839 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:54.839 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:54.839 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:27:54.839 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:54.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:27:54.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:54.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:54.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:27:54.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:27:54.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:54.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:54.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:54.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:54.945 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:54.946 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:54.947 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:54.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:54.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:54.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:55.036 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:55.036 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:55.036 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:55.036 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:55.037 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:55.048 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:55.049 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:27:55.049 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:27:55.049 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:27:55.050 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:55.050 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:55.050 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:55.051 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:27:55.051 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:55.051 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:27:55.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:55.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:55.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:27:55.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:27:55.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 0 = 0 ']' 2021-07-26T06:27:55.151 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: expr 0 + 1 2021-07-26T06:27:55.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: zerocount=1 2021-07-26T06:27:55.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 1 -gt 3 ']' 2021-07-26T06:27:55.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 704 + 0 2021-07-26T06:27:55.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=704 2021-07-26T06:27:55.153 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:27:55.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:27:59.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:27:59.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:27:59.157 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:27:59.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:27:59.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:59.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:59.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:59.161 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:59.162 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:59.163 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:59.163 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:59.163 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:59.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:59.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:59.266 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:59.266 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:59.266 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:59.267 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:59.267 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:59.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:59.277 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:27:59.277 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:27:59.278 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:27:59.278 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:59.278 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:59.278 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:59.279 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:27:59.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:59.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:27:59.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:59.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:59.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:27:59.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:27:59.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:59.383 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:59.383 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:59.384 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:59.384 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:59.384 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:59.384 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:59.384 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:59.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:59.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:59.477 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:59.478 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:59.478 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:59.478 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:59.478 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:59.488 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:59.489 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:27:59.489 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:27:59.490 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:27:59.490 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:59.490 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:59.491 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:59.491 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:27:59.491 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:59.492 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:27:59.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:59.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:59.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:27:59.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:27:59.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:59.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:59.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:59.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:59.599 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:59.599 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:59.600 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:59.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:59.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:59.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:59.690 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:59.691 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:59.691 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:59.691 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:59.691 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:59.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:59.701 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:27:59.701 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:27:59.701 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:27:59.701 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:59.702 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:59.702 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:59.702 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:27:59.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:59.703 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:27:59.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:27:59.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:27:59.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:27:59.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:27:59.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:27:59.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:59.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:59.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:59.807 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:59.807 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:59.807 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:59.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:59.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:59.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:27:59.899 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:27:59.900 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:27:59.901 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:27:59.901 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:27:59.901 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:27:59.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:27:59.909 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:27:59.909 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:27:59.910 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:27:59.910 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:27:59.911 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:27:59.911 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:27:59.911 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:27:59.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:27:59.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:28:00.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:00.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:00.013 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:00.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:00.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:00.016 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:28:00.016 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:28:00.016 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:28:00.017 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:00.017 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:00.017 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:00.017 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:28:00.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:00.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:28:00.109 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:00.109 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:00.109 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:00.110 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:00.110 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:00.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:00.119 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:28:00.120 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:28:00.120 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:28:00.120 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:00.121 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:00.121 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:00.122 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:28:00.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:00.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:28:00.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:00.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:00.223 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:00.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:00.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:00.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:28:00.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:28:00.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:28:00.226 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:00.227 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:00.227 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:00.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:28:00.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:00.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:28:00.320 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:00.320 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:00.321 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:00.321 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:00.321 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:00.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:00.331 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:28:00.331 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:28:00.331 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:28:00.331 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:00.331 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:00.332 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:00.332 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:28:00.332 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:00.332 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:28:00.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:00.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:00.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:00.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:00.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 0 = 0 ']' 2021-07-26T06:28:00.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: expr 1 + 1 2021-07-26T06:28:00.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: zerocount=2 2021-07-26T06:28:00.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 2 -gt 3 ']' 2021-07-26T06:28:00.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 704 + 0 2021-07-26T06:28:00.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=704 2021-07-26T06:28:00.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:28:00.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:28:04.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:28:04.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:28:04.443 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:28:04.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:28:04.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:04.446 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:28:04.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:28:04.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:28:04.447 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:04.447 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:04.448 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:04.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:28:04.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:04.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:28:04.550 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:04.551 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:04.551 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:04.551 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:04.551 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:04.563 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:04.563 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:28:04.564 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:28:04.564 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:28:04.565 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:04.565 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:04.565 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:04.566 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:28:04.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:04.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:28:04.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:04.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:04.665 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:04.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:04.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:04.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:28:04.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:28:04.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:28:04.670 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:04.670 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:04.670 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:04.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:28:04.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:04.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:28:04.762 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:04.763 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:04.763 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:04.763 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:04.763 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:04.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:04.773 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:28:04.773 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:28:04.773 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:28:04.773 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:04.773 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:04.774 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:04.774 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:28:04.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:04.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:28:04.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:04.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:04.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:04.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:04.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:04.877 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:28:04.878 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:28:04.878 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:28:04.878 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:04.878 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:04.879 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:04.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:28:04.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:04.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:28:04.972 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:04.972 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:04.973 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:04.973 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:04.973 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:04.982 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:04.982 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:28:04.983 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:28:04.983 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:28:04.983 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:04.983 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:04.984 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:04.984 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:28:04.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:04.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:28:05.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:05.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:05.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:05.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:05.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:05.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:28:05.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:28:05.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:28:05.087 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:05.088 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:05.088 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:05.088 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:28:05.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:05.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:28:05.181 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:05.181 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:05.181 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:05.182 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:05.182 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:05.191 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:05.191 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:28:05.192 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:28:05.192 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:28:05.192 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:05.192 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:05.192 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:05.193 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:28:05.193 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:05.193 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:28:05.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:05.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:05.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:05.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:05.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:05.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:28:05.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:28:05.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:28:05.297 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:05.298 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:05.298 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:05.298 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:28:05.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:05.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:28:05.391 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:05.391 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:05.392 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:05.392 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:05.392 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:05.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:05.402 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:28:05.402 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:28:05.402 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:28:05.402 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:05.403 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:05.403 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:05.403 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:28:05.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:05.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:28:05.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:05.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:05.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:05.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:05.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:05.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:28:05.507 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:28:05.507 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:28:05.507 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:05.507 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:05.508 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:05.508 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:28:05.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:05.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:28:05.600 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:05.600 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:05.600 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:05.601 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:05.601 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:05.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:05.610 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:28:05.611 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:28:05.611 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:28:05.611 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:05.611 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:05.612 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:05.612 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:28:05.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:05.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:28:05.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:05.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:05.713 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:05.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:05.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 0 = 0 ']' 2021-07-26T06:28:05.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: expr 2 + 1 2021-07-26T06:28:05.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: zerocount=3 2021-07-26T06:28:05.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 3 -gt 3 ']' 2021-07-26T06:28:05.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 704 + 0 2021-07-26T06:28:05.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=704 2021-07-26T06:28:05.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2021-07-26T06:28:05.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2021-07-26T06:28:09.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2021-07-26T06:28:09.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2021-07-26T06:28:09.722 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2021-07-26T06:28:09.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2021-07-26T06:28:09.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:09.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:28:09.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:28:09.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:28:09.727 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:09.727 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:09.727 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:09.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:28:09.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:09.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:28:09.829 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:09.829 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:09.830 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:09.830 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:09.830 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:09.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:09.841 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2021-07-26T06:28:09.841 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:28:09.841 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:28:09.841 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:09.842 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:09.842 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:09.842 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.0.asok 2021-07-26T06:28:09.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:09.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.0.asok dump_scrub_reservations 2021-07-26T06:28:09.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:09.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:09.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:09.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:09.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:09.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:28:09.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:28:09.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:28:09.951 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:09.951 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:09.951 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:09.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:28:09.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:09.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:28:10.041 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:10.042 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:10.042 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:10.043 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:10.043 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:10.052 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:10.052 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2021-07-26T06:28:10.053 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:28:10.053 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:28:10.053 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:10.053 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:10.054 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:10.054 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.1.asok 2021-07-26T06:28:10.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:10.055 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.1.asok dump_scrub_reservations 2021-07-26T06:28:10.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:10.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:10.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:10.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:10.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:10.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:28:10.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:28:10.159 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:28:10.159 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:10.159 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:10.159 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:10.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:28:10.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:10.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:28:10.250 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:10.250 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:10.250 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:10.251 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:10.251 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:10.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:10.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2021-07-26T06:28:10.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:28:10.263 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:28:10.263 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:10.264 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:10.264 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:10.265 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.2.asok 2021-07-26T06:28:10.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:10.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.2.asok dump_scrub_reservations 2021-07-26T06:28:10.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:10.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:10.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:10.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:10.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:10.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:28:10.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:28:10.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:28:10.366 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:10.367 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:10.367 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:10.367 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:28:10.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:10.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:28:10.458 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:10.458 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:10.459 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:10.459 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:10.459 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:10.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:10.469 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2021-07-26T06:28:10.469 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-07-26T06:28:10.469 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-07-26T06:28:10.470 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:10.470 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:10.470 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:10.470 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.3.asok 2021-07-26T06:28:10.471 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:10.471 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.3.asok dump_scrub_reservations 2021-07-26T06:28:10.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:10.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:10.572 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:10.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:10.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:10.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:28:10.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:28:10.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:28:10.575 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:10.575 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:10.575 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:10.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:28:10.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:10.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:28:10.668 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:10.669 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:10.669 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:10.669 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:10.670 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:10.678 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:10.679 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2021-07-26T06:28:10.679 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2021-07-26T06:28:10.679 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2021-07-26T06:28:10.680 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:10.680 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:10.680 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:10.680 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.4.asok 2021-07-26T06:28:10.681 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:10.681 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.4.asok dump_scrub_reservations 2021-07-26T06:28:10.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:10.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:10.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:10.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:10.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:28:10.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:28:10.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:28:10.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:28:10.784 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:10.784 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:10.784 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:10.784 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:28:10.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:10.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:28:10.876 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:28:10.877 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_local": 0, 2021-07-26T06:28:10.877 INFO:tasks.workunit.client.0.smithi071.stdout: "scrubs_remote": 0, 2021-07-26T06:28:10.877 INFO:tasks.workunit.client.0.smithi071.stdout: "osd_max_scrubs": 4 2021-07-26T06:28:10.877 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:28:10.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2021-07-26T06:28:10.888 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2021-07-26T06:28:10.888 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2021-07-26T06:28:10.888 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2021-07-26T06:28:10.889 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:10.889 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:10.890 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:10.890 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53231/ceph-osd.5.asok 2021-07-26T06:28:10.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2021-07-26T06:28:10.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.53231/ceph-osd.5.asok dump_scrub_reservations 2021-07-26T06:28:10.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2021-07-26T06:28:10.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2021-07-26T06:28:10.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2021-07-26T06:28:10.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2021-07-26T06:28:10.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 0 = 0 ']' 2021-07-26T06:28:10.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: expr 3 + 1 2021-07-26T06:28:10.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: zerocount=4 2021-07-26T06:28:10.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 4 -gt 3 ']' 2021-07-26T06:28:10.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: break 2021-07-26T06:28:10.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: seq 0 5 2021-07-26T06:28:10.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: for i in $(seq 0 5) 2021-07-26T06:28:10.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:28:10.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: grep scrubbing 2021-07-26T06:28:11.254 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:28:11.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: sleep 2 2021-07-26T06:28:13.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: for i in $(seq 0 5) 2021-07-26T06:28:13.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:28:13.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: grep scrubbing 2021-07-26T06:28:13.562 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:28:13.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: sleep 2 2021-07-26T06:28:15.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: for i in $(seq 0 5) 2021-07-26T06:28:15.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:28:15.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: grep scrubbing 2021-07-26T06:28:15.841 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:28:15.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: sleep 2 2021-07-26T06:28:17.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: for i in $(seq 0 5) 2021-07-26T06:28:17.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:28:17.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: grep scrubbing 2021-07-26T06:28:18.125 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:28:18.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: sleep 2 2021-07-26T06:28:20.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: for i in $(seq 0 5) 2021-07-26T06:28:20.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:28:20.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: grep scrubbing 2021-07-26T06:28:20.406 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:28:20.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: sleep 2 2021-07-26T06:28:22.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: for i in $(seq 0 5) 2021-07-26T06:28:22.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: ceph pg dump pgs 2021-07-26T06:28:22.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: grep scrubbing 2021-07-26T06:28:22.684 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:28:22.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: sleep 2 2021-07-26T06:28:24.695 INFO:tasks.workunit.client.0.smithi071.stdout:704 total reservations seen 2021-07-26T06:28:24.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:153: TEST_recover_unexpected: echo 704 total reservations seen 2021-07-26T06:28:24.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:158: TEST_recover_unexpected: expr 16 '*' 3 '*' 3 2021-07-26T06:28:24.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:158: TEST_recover_unexpected: actual_reservations=144 2021-07-26T06:28:24.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:159: TEST_recover_unexpected: '[' 704 -lt 144 ']' 2021-07-26T06:28:24.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:164: TEST_recover_unexpected: return 0 2021-07-26T06:28:24.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:45: run: teardown td/osd-scrub-dump 2021-07-26T06:28:24.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2021-07-26T06:28:24.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:28:24.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2021-07-26T06:28:24.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:28:24.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:28:24.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:28:24.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:28:24.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:28:24.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:28:24.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:28:24.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:28:24.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:28:24.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:28:24.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:28:24.815 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:28:24.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:28:24.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:28:24.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:28:24.817 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:28:24.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:28:24.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:28:24.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2021-07-26T06:28:24.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:28:24.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:24.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:24.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53231 2021-07-26T06:28:24.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:28:24.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:28:24.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:28:24.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-scrub-dump 0 2021-07-26T06:28:24.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2021-07-26T06:28:24.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:28:24.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2021-07-26T06:28:24.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:28:24.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:28:24.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:28:24.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:28:24.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:28:24.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:28:24.954 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:28:24.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:28:24.955 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:28:24.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:28:24.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:28:24.956 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:28:24.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:28:24.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:28:24.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:28:24.956 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:28:24.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:28:24.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:28:24.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2021-07-26T06:28:24.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:28:24.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:24.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53231 2021-07-26T06:28:24.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53231 2021-07-26T06:28:24.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:28:24.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:28:24.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:28:24.961 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:28:24.962 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:28:24.996 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2021-07-26T06:28:24.997 DEBUG:teuthology.orchestra.run.smithi071:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2021-07-26T06:28:25.057 INFO:tasks.workunit.client.0.smithi071.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2021-07-26T06:28:25.057 INFO:tasks.workunit.client.0.smithi071.stderr:++ TIMEOUT=300 2021-07-26T06:28:25.057 INFO:tasks.workunit.client.0.smithi071.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2021-07-26T06:28:25.058 INFO:tasks.workunit.client.0.smithi071.stderr:++ MAX_TIMEOUT=15 2021-07-26T06:28:25.058 INFO:tasks.workunit.client.0.smithi071.stderr:++ PG_NUM=4 2021-07-26T06:28:25.058 INFO:tasks.workunit.client.0.smithi071.stderr:++ TMPDIR=/tmp 2021-07-26T06:28:25.058 INFO:tasks.workunit.client.0.smithi071.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2021-07-26T06:28:25.059 INFO:tasks.workunit.client.0.smithi071.stderr:++ TESTDIR=/home/ubuntu/cephtest 2021-07-26T06:28:25.059 INFO:tasks.workunit.client.0.smithi071.stderr:++ type xmlstarlet 2021-07-26T06:28:25.060 INFO:tasks.workunit.client.0.smithi071.stderr:++ XMLSTARLET=xmlstarlet 2021-07-26T06:28:25.060 INFO:tasks.workunit.client.0.smithi071.stderr:+++ uname 2021-07-26T06:28:25.060 INFO:tasks.workunit.client.0.smithi071.stderr:++ '[' Linux = FreeBSD ']' 2021-07-26T06:28:25.060 INFO:tasks.workunit.client.0.smithi071.stderr:++ SED=sed 2021-07-26T06:28:25.061 INFO:tasks.workunit.client.0.smithi071.stderr:++ AWK=awk 2021-07-26T06:28:25.061 INFO:tasks.workunit.client.0.smithi071.stderr:+++ stty -a 2021-07-26T06:28:25.061 INFO:tasks.workunit.client.0.smithi071.stderr:+++ head -1 2021-07-26T06:28:25.061 INFO:tasks.workunit.client.0.smithi071.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2021-07-26T06:28:25.061 INFO:tasks.workunit.client.0.smithi071.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:28:25.063 INFO:tasks.workunit.client.0.smithi071.stderr:++ termwidth= 2021-07-26T06:28:25.063 INFO:tasks.workunit.client.0.smithi071.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2021-07-26T06:28:25.063 INFO:tasks.workunit.client.0.smithi071.stderr:++ DIFFCOLOPTS='-y ' 2021-07-26T06:28:25.064 INFO:tasks.workunit.client.0.smithi071.stderr:++ KERNCORE=kernel.core_pattern 2021-07-26T06:28:25.064 INFO:tasks.workunit.client.0.smithi071.stderr:++ EXTRA_OPTS= 2021-07-26T06:28:25.064 INFO:tasks.workunit.client.0.smithi071.stderr:++ test '' = TESTS 2021-07-26T06:28:25.065 INFO:tasks.workunit.client.0.smithi071.stderr:++ uname 2021-07-26T06:28:25.066 INFO:tasks.workunit.client.0.smithi071.stderr:+ '[' Linux = FreeBSD ']' 2021-07-26T06:28:25.066 INFO:tasks.workunit.client.0.smithi071.stderr:+ use_ec_overwrite=true 2021-07-26T06:28:25.066 INFO:tasks.workunit.client.0.smithi071.stderr:+ getjson=no 2021-07-26T06:28:25.066 INFO:tasks.workunit.client.0.smithi071.stderr:+ jqfilter='def walk(f): 2021-07-26T06:28:25.067 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T06:28:25.067 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T06:28:25.067 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T06:28:25.067 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T06:28:25.067 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T06:28:25.068 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T06:28:25.068 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T06:28:25.068 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T06:28:25.068 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T06:28:25.069 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T06:28:25.069 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T06:28:25.069 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2021-07-26T06:28:25.069 INFO:tasks.workunit.client.0.smithi071.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T06:28:25.072 INFO:tasks.workunit.client.0.smithi071.stderr:+ main osd-scrub-repair 2021-07-26T06:28:25.073 INFO:tasks.workunit.client.0.smithi071.stderr:+ local dir=td/osd-scrub-repair 2021-07-26T06:28:25.073 INFO:tasks.workunit.client.0.smithi071.stderr:+ shift 2021-07-26T06:28:25.073 INFO:tasks.workunit.client.0.smithi071.stderr:+ shopt -s -o xtrace 2021-07-26T06:28:25.073 INFO:tasks.workunit.client.0.smithi071.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:28:25.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-07-26T06:28:25.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-07-26T06:28:25.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:28:25.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:28:25.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:28:25.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:28:25.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:28:25.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:28:25.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-scrub-repair 2021-07-26T06:28:25.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:51: run: local dir=td/osd-scrub-repair 2021-07-26T06:28:25.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: shift 2021-07-26T06:28:25.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:54: run: export CEPH_MON=127.0.0.1:7107 2021-07-26T06:28:25.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:54: run: CEPH_MON=127.0.0.1:7107 2021-07-26T06:28:25.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_ARGS 2021-07-26T06:28:25.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: uuidgen 2021-07-26T06:28:25.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: CEPH_ARGS+='--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none ' 2021-07-26T06:28:25.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2021-07-26T06:28:25.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2021-07-26T06:28:25.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:60: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:28:25.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: set 2021-07-26T06:28:25.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:28:25.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: local 'funcs=TEST_allow_repair_during_recovery 2021-07-26T06:28:25.093 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_auto_repair_bluestore_basic 2021-07-26T06:28:25.093 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_auto_repair_bluestore_failed 2021-07-26T06:28:25.094 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_auto_repair_bluestore_failed_norecov 2021-07-26T06:28:25.094 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_auto_repair_bluestore_scrub 2021-07-26T06:28:25.094 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_auto_repair_bluestore_tag 2021-07-26T06:28:25.095 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_auto_repair_erasure_coded_appends 2021-07-26T06:28:25.096 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_auto_repair_erasure_coded_overwrites 2021-07-26T06:28:25.096 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_corrupt_and_repair_jerasure_appends 2021-07-26T06:28:25.097 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2021-07-26T06:28:25.097 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_corrupt_and_repair_lrc_appends 2021-07-26T06:28:25.097 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_corrupt_and_repair_lrc_overwrites 2021-07-26T06:28:25.097 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_corrupt_and_repair_replicated 2021-07-26T06:28:25.098 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_corrupt_scrub_erasure_appends 2021-07-26T06:28:25.098 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_corrupt_scrub_erasure_overwrites 2021-07-26T06:28:25.098 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_corrupt_scrub_replicated 2021-07-26T06:28:25.098 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_corrupt_snapset_scrub_rep 2021-07-26T06:28:25.099 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_list_missing_erasure_coded_appends 2021-07-26T06:28:25.099 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_list_missing_erasure_coded_overwrites 2021-07-26T06:28:25.099 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_periodic_scrub_replicated 2021-07-26T06:28:25.099 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_repair_stats 2021-07-26T06:28:25.100 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_repair_stats_ec 2021-07-26T06:28:25.100 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_request_scrub_priority 2021-07-26T06:28:25.100 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_scrub_warning 2021-07-26T06:28:25.100 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_skip_non_repair_during_recovery 2021-07-26T06:28:25.101 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_unfound_erasure_coded_appends 2021-07-26T06:28:25.101 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_unfound_erasure_coded_overwrites' 2021-07-26T06:28:25.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:28:25.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:28:25.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:28:25.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:28:25.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:28:25.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:28:25.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:28:25.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:28:25.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:28:25.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:28:25.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:28:25.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:28:25.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:28:25.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:28:25.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:28:25.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:28:25.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:28:25.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:28:25.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:28:25.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:28:25.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:28:25.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:28:25.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:28:25.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:28:25.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:28:25.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:28:25.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:28:25.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:25.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:28:25.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:28:25.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:28:25.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:28:25.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:28:25.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:28:25.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:25.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:28:25.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:28:25.111 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:28:25.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:28:25.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:28:25.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:28:25.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:28:25.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2021-07-26T06:28:25.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:113: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2021-07-26T06:28:25.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local poolname=rbd 2021-07-26T06:28:25.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:116: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2021-07-26T06:28:25.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:28:25.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:28:25.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:28:25.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:28:25.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:28:25.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2021-07-26T06:28:25.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:28:25.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:28:25.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:28:25.460 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:28:25.460 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:25.460 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:28:25.460 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:28:25.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2021-07-26T06:28:25.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:28:25.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:28:25.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:28:25.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:28:25.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:28:25.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:28:25.755 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:28:25.755 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:28:25.756 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:28:25.756 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:25.756 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:25.756 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:28:25.757 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:28:25.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:28:25.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:28:25.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:28:25.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:28:25.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:28:25.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:28:25.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:28:25.855 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:28:25.855 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:28:25.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:28:25.856 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:28:25.856 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:25.856 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:28:25.857 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:28:25.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:28:25.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:28:25.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2021-07-26T06:28:25.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:28:25.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:28:25.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:28:25.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:28:25.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:28:25.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:28:26.788 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:28:26.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:28:26.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:28:26.789 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:28:26.789 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:26.790 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:28:26.790 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:28:26.790 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:28:26.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:28:26.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2021-07-26T06:28:26.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:28:26.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:28:26.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:28:26.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:28:26.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:28:26.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:28:26.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:28:26.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:28:26.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:28:26.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:28:26.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:28:26.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:28:26.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:28:26.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:28:26.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:28:26.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:28:26.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:28:26.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:28:26.826 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:28:26.826 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:26.827 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:28:26.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:28:26.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:28:26.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:28:26.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:28:26.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:28:26.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:28:26.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:28:26.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:28:26.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:28:26.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:28:26.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:28:26.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2021-07-26T06:28:26.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:28:26.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:28:26.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=566ac580-56fa-4fa4-943c-26ae58e101e8 2021-07-26T06:28:26.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 566ac580-56fa-4fa4-943c-26ae58e101e8' 2021-07-26T06:28:26.831 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 566ac580-56fa-4fa4-943c-26ae58e101e8 2021-07-26T06:28:26.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:28:26.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAKVv5gjQeMMRAAI+EtkUouEmn2dGCU0KO+Cw== 2021-07-26T06:28:26.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAKVv5gjQeMMRAAI+EtkUouEmn2dGCU0KO+Cw=="}' 2021-07-26T06:28:26.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 566ac580-56fa-4fa4-943c-26ae58e101e8 -i td/osd-scrub-repair/0/new.json 2021-07-26T06:28:27.093 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:28:27.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:28:27.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQAKVv5gjQeMMRAAI+EtkUouEmn2dGCU0KO+Cw== --osd-uuid 566ac580-56fa-4fa4-943c-26ae58e101e8 2021-07-26T06:28:27.131 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:27.131+0000 7fd6ebad20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:27.132 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:27.132+0000 7fd6ebad20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:27.133 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:27.133+0000 7fd6ebad20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:27.134 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:27.134+0000 7fd6ebad20c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:28:27.134 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:27.134+0000 7fd6ebad20c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:28:27.134 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:27.134+0000 7fd6ebad20c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:28:27.227 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:27.227+0000 7fd6ebad20c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T06:28:33.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:28:33.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:28:33.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:28:33.200 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:28:33.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:28:33.567 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:28:33.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:28:33.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2021-07-26T06:28:33.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:28:33.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:28:33.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:28:33.591 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:33.592+0000 7f3b4d8ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:33.666 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:33.667+0000 7f3b4d8ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:33.668 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:33.668+0000 7f3b4d8ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:33.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:28:33.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:28:33.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:28:33.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:28:33.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:28:33.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:33.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:28:33.845 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:28:33.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:33.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:28:34.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:34.183 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:34.185+0000 7f3b4d8ff0c0 -1 Falling back to public interface 2021-07-26T06:28:35.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:35.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:35.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:28:35.109 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:28:35.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:35.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:28:35.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:36.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:36.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:36.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:28:36.383 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:28:36.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:36.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:28:36.583 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:36.584+0000 7f3b4d8ff0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:28:36.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:37.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:37.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:37.684 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:28:37.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:28:37.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:37.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:28:37.955 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/130036,v1:127.0.0.1:6803/130036] [v2:127.0.0.1:6804/130036,v1:127.0.0.1:6805/130036] exists,up 566ac580-56fa-4fa4-943c-26ae58e101e8 2021-07-26T06:28:37.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:28:37.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:28:37.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:28:37.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:121: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2021-07-26T06:28:37.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:28:37.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:28:37.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:28:37.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:28:37.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:28:37.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:28:37.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:28:37.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:28:37.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:28:37.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:28:37.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:28:37.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:28:37.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:28:37.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:28:37.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:28:37.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:28:37.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:28:37.965 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:28:37.965 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:28:37.965 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:37.965 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:28:37.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:28:37.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:28:37.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:28:37.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:28:37.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:28:37.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:28:37.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:28:37.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:28:37.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:28:37.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:28:37.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:28:37.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2021-07-26T06:28:37.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:28:37.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:28:37.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1c45873d-aab6-4681-8f66-14b700ce0821 2021-07-26T06:28:37.969 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 1c45873d-aab6-4681-8f66-14b700ce0821 2021-07-26T06:28:37.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1c45873d-aab6-4681-8f66-14b700ce0821' 2021-07-26T06:28:37.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:28:37.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVVv5gfLwKOhAA1yl2w/Zha+I8FKhOuhPJWw== 2021-07-26T06:28:37.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVVv5gfLwKOhAA1yl2w/Zha+I8FKhOuhPJWw=="}' 2021-07-26T06:28:37.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1c45873d-aab6-4681-8f66-14b700ce0821 -i td/osd-scrub-repair/1/new.json 2021-07-26T06:28:38.407 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:28:38.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:28:38.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQAVVv5gfLwKOhAA1yl2w/Zha+I8FKhOuhPJWw== --osd-uuid 1c45873d-aab6-4681-8f66-14b700ce0821 2021-07-26T06:28:38.446 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:38.447+0000 7fb371f5b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:38.453 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:38.453+0000 7fb371f5b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:38.453 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:38.454+0000 7fb371f5b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:38.454 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:38.454+0000 7fb371f5b0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:28:38.454 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:38.454+0000 7fb371f5b0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:28:38.454 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:38.454+0000 7fb371f5b0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:28:38.527 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:38.528+0000 7fb371f5b0c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T06:28:44.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:28:44.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:28:44.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:28:44.309 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:28:44.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:28:44.650 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:28:44.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:28:44.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2021-07-26T06:28:44.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:28:44.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:28:44.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:28:44.674 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:44.675+0000 7fd9c57d60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:44.675 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:44.677+0000 7fd9c57d60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:44.676 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:44.677+0000 7fd9c57d60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:44.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:28:44.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:28:44.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:28:44.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:28:44.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:28:44.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:44.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:28:44.921 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:28:44.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:44.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:28:45.195 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:45.197+0000 7fd9c57d60c0 -1 Falling back to public interface 2021-07-26T06:28:45.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:46.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:46.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:46.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:28:46.228 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:28:46.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:46.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:28:46.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:47.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:47.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:47.506 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:28:47.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:28:47.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:47.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:28:47.642 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:28:47.643+0000 7fd9c57d60c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:28:47.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:48.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:48.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:48.892 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:28:48.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:28:48.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:48.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:28:49.251 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/131417,v1:127.0.0.1:6811/131417] [v2:127.0.0.1:6812/131417,v1:127.0.0.1:6813/131417] exists,up 1c45873d-aab6-4681-8f66-14b700ce0821 2021-07-26T06:28:49.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:28:49.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:28:49.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:28:49.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:124: TEST_allow_repair_during_recovery: create_rbd_pool 2021-07-26T06:28:49.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:28:49.551 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T06:28:49.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:28:49.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:28:50.052 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T06:28:50.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:28:51.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:28:52.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: wait_for_clean 2021-07-26T06:28:52.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:28:52.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:28:52.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:28:52.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:28:52.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:28:52.890 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:28:52.890 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:28:52.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:28:52.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:28:52.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:28:52.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:28:52.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:28:52.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:28:52.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:28:52.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:28:53.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:28:53.260 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T06:28:53.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:28:53.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:28:53.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:28:53.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836485 2021-07-26T06:28:53.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836485 2021-07-26T06:28:53.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836485' 2021-07-26T06:28:53.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:28:53.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:28:53.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672962 2021-07-26T06:28:53.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672962 2021-07-26T06:28:53.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836485 1-42949672962' 2021-07-26T06:28:53.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:28:53.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836485 2021-07-26T06:28:53.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:28:53.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:28:53.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836485 2021-07-26T06:28:53.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:28:53.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836485 2021-07-26T06:28:53.587 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836485 2021-07-26T06:28:53.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2021-07-26T06:28:53.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:28:53.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836484 -lt 21474836485 2021-07-26T06:28:53.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:28:54.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:28:54.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:28:55.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836484 -lt 21474836485 2021-07-26T06:28:55.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:28:56.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:28:56.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:28:56.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836485 -lt 21474836485 2021-07-26T06:28:56.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:28:56.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672962 2021-07-26T06:28:56.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:28:56.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:28:56.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672962 2021-07-26T06:28:56.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:28:56.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672962 2021-07-26T06:28:56.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2021-07-26T06:28:56.460 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672962 2021-07-26T06:28:56.460 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:28:56.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672963 -lt 42949672962 2021-07-26T06:28:56.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:28:56.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:28:56.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:28:57.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:28:57.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:28:57.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:28:57.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:28:57.055 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:28:57.055 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:28:57.055 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:28:57.055 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:28:57.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:28:57.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:28:57.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:28:57.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:28:57.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:28:57.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:28:57.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:28:57.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:127: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2021-07-26T06:28:57.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:28:57.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=rbd 2021-07-26T06:28:57.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2021-07-26T06:28:57.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:28:57.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:28:57.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:28:58.120 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:28:58.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:28:58.664 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:28:58.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:28:58.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:28:58.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T06:28:58.957 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2021-07-26T06:28:58.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=rbd 2021-07-26T06:28:58.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:28:58.958 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary rbd SOMETHING 2021-07-26T06:28:58.959 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=rbd 2021-07-26T06:28:58.959 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:28:58.959 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map rbd SOMETHING 2021-07-26T06:28:58.959 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:28:59.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:28:59.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map rbd SOMETHING 2021-07-26T06:28:59.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:28:59.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2021-07-26T06:28:59.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2021-07-26T06:28:59.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=rbd 2021-07-26T06:28:59.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=0 2021-07-26T06:28:59.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:28:59.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:28:59.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:28:59.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:28:59.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:28:59.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:28:59.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:28:59.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:28:59.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:28:59.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:28:59.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:28:59.502 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:28:59.503 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:28:59.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:28:59.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:28:59.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:28:59.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:28:59.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:28:59.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:28:59.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:28:59.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:28:59.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:28:59.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:28:59.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2021-07-26T06:29:01.234 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:29:01.234 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:29:01.234 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:29:01.235 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:29:01.235 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:eb822e21:::SOMETHING:head# 2021-07-26T06:29:01.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T06:29:01.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:29:01.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:29:01.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:29:01.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:29:01.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:29:01.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:29:01.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:29:01.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:29:01.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:29:01.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:29:01.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:29:01.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:29:01.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:29:01.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:29:01.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:29:01.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:29:01.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:01.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:01.759 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:01.759 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:01.759 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:29:01.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:29:01.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:29:01.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:29:01.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:29:01.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:29:01.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:29:01.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:29:01.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:29:01.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:29:01.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:29:01.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:29:01.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:29:01.762 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:29:01.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:29:01.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:29:01.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:29:01.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:29:01.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:29:01.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:29:01.778 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:01.779+0000 7f408acfa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:01.780 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:01.781+0000 7f408acfa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:01.781 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:01.782+0000 7f408acfa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:02.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:29:02.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:29:02.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:29:02.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:29:02.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:29:02.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:02.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:29:02.036 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:29:02.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:02.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:02.296 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:02.298+0000 7f408acfa0c0 -1 Falling back to public interface 2021-07-26T06:29:02.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:03.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:03.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:03.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:29:03.328 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:29:03.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:03.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:03.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:04.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:04.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:04.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:29:04.632 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:29:04.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:04.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:04.802 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:04.803+0000 7f408acfa0c0 -1 osd.0 22 log_to_monitors {default=true} 2021-07-26T06:29:04.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:05.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:05.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:05.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:29:05.917 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:29:05.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:05.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:06.314 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/132919,v1:127.0.0.1:6803/132919] [v2:127.0.0.1:6804/132919,v1:127.0.0.1:6805/132919] exists,up 566ac580-56fa-4fa4-943c-26ae58e101e8 2021-07-26T06:29:06.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:29:06.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:29:06.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:29:06.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:29:06.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:29:06.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:29:06.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:29:06.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:29:06.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:29:06.317 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:29:06.317 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:29:06.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:29:06.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:29:06.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:29:06.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:29:06.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:29:06.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:29:06.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:29:06.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:29:06.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:29:06.695 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T06:29:06.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:29:06.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:29:06.695 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:29:06.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=111669149698 2021-07-26T06:29:06.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 111669149698 2021-07-26T06:29:06.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149698' 2021-07-26T06:29:06.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:29:06.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:29:06.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672966 2021-07-26T06:29:06.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672966 2021-07-26T06:29:06.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149698 1-42949672966' 2021-07-26T06:29:06.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:29:06.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-111669149698 2021-07-26T06:29:06.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:29:06.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:29:06.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-111669149698 2021-07-26T06:29:06.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:29:06.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=111669149698 2021-07-26T06:29:06.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2021-07-26T06:29:06.946 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 111669149698 2021-07-26T06:29:06.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:29:07.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 111669149698 2021-07-26T06:29:07.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:29:08.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:29:08.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:29:08.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149698 -lt 111669149698 2021-07-26T06:29:08.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:29:08.490 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672966 2021-07-26T06:29:08.490 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:29:08.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:29:08.491 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672966 2021-07-26T06:29:08.491 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:29:08.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672966 2021-07-26T06:29:08.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-07-26T06:29:08.492 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672966 2021-07-26T06:29:08.493 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:29:08.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672966 -lt 42949672966 2021-07-26T06:29:08.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:29:08.800 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:29:08.800 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:29:09.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:29:09.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:29:09.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:29:09.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:29:09.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:29:09.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:29:09.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:29:09.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:29:09.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:29:09.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:29:09.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:29:09.471 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:29:09.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:29:09.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:29:09.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:29:09.797 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg rbd SOMETHING 2021-07-26T06:29:09.797 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=rbd 2021-07-26T06:29:09.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:29:09.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map rbd SOMETHING 2021-07-26T06:29:09.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:29:10.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=1.3 2021-07-26T06:29:10.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 1.3 2021-07-26T06:29:10.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=1.3 2021-07-26T06:29:10.067 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 1.3 2021-07-26T06:29:10.067 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:29:10.067 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:29:10.067 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:29:10.068 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:29:10.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:28:49.913333+0000 2021-07-26T06:29:10.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 1.3 2021-07-26T06:29:10.619 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 1.3 on osd.1 to repair 2021-07-26T06:29:10.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 1.3 2021-07-26T06:28:49.913333+0000 2021-07-26T06:29:10.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.3 2021-07-26T06:29:10.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:28:49.913333+0000 2021-07-26T06:29:10.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:29:10.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:29:10.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:29:10.630 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:29:10.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:29:10.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:29:10.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:29:10.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:29:10.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:28:49.913333+0000 '>' 2021-07-26T06:28:49.913333+0000 2021-07-26T06:29:10.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:29:11.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:29:11.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:29:11.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:29:11.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:29:11.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:29:11.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:29:11.892 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:29:12.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:28:49.913333+0000 '>' 2021-07-26T06:28:49.913333+0000 2021-07-26T06:29:12.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:29:13.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:29:13.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:29:13.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:29:13.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:29:13.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:29:13.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:29:13.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:29:13.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:28:49.913333+0000 '>' 2021-07-26T06:28:49.913333+0000 2021-07-26T06:29:13.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:29:14.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:29:14.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:29:14.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:29:14.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:29:14.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:29:14.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:29:14.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:29:14.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:28:49.913333+0000 '>' 2021-07-26T06:28:49.913333+0000 2021-07-26T06:29:14.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:29:15.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:29:15.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:29:15.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:29:15.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:29:15.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:29:15.715 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:29:15.715 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:29:15.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:28:49.913333+0000 '>' 2021-07-26T06:28:49.913333+0000 2021-07-26T06:29:15.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:29:16.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:29:16.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:29:16.992 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:29:16.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:29:16.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:29:16.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:29:16.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:29:17.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:28:49.913333+0000 '>' 2021-07-26T06:28:49.913333+0000 2021-07-26T06:29:17.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:29:18.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:29:18.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:29:18.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:29:18.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:29:18.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:29:18.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:29:18.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:29:18.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:28:49.913333+0000 '>' 2021-07-26T06:28:49.913333+0000 2021-07-26T06:29:18.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:29:19.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:29:19.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:29:19.533 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:29:19.533 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:29:19.533 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:29:19.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:29:19.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:29:19.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:29:15.768300+0000 '>' 2021-07-26T06:28:49.913333+0000 2021-07-26T06:29:19.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:29:19.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:29:19.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:29:19.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:29:19.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:29:19.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:29:19.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:29:19.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:29:19.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:29:19.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:29:19.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:29:19.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:29:19.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:29:19.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:29:19.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:29:19.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:29:19.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:29:19.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:29:19.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:29:19.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:29:19.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:29:19.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:29:19.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:29:19.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:29:19.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2021-07-26T06:29:21.634 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2021-07-26T06:29:21.635 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:29:21.635 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:29:21.635 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:29:21.636 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:29:21.636 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:29:21.636 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:29:21.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:29:21.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:29:21.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:29:21.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:29:21.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:29:21.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:29:21.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:29:21.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:29:21.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:29:21.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:29:21.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:29:21.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:29:21.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:29:21.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:29:21.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:29:21.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:29:21.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:29:21.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:21.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:21.921 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:21.922 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:21.922 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:29:21.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:29:21.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:29:21.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:29:21.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:29:21.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:29:21.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:29:21.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:29:21.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:29:21.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:29:21.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:29:21.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:29:21.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:29:21.925 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:29:21.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:29:21.926 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:29:21.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:29:21.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:29:21.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:29:21.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:29:21.943 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:21.939+0000 7f9d3842b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:21.960 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:21.960+0000 7f9d3842b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:21.961 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:21.961+0000 7f9d3842b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:22.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:29:22.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:29:22.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:29:22.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:29:22.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:29:22.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:22.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:29:22.197 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:29:22.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:22.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:29:22.480 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:22.481+0000 7f9d3842b0c0 -1 Falling back to public interface 2021-07-26T06:29:22.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:23.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:23.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:23.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:29:23.506 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:29:23.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:23.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:29:23.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:24.772 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:24.772+0000 7f9d3842b0c0 -1 osd.1 27 log_to_monitors {default=true} 2021-07-26T06:29:24.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:24.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:24.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:29:24.785 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:29:24.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:24.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:29:25.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:26.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:26.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:26.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:29:26.087 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:29:26.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:26.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:29:26.388 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 31 up_thru 31 down_at 28 last_clean_interval [10,27) [v2:127.0.0.1:6810/134503,v1:127.0.0.1:6811/134503] [v2:127.0.0.1:6812/134503,v1:127.0.0.1:6813/134503] exists,up 1c45873d-aab6-4681-8f66-14b700ce0821 2021-07-26T06:29:26.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:29:26.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:29:26.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:29:26.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:29:26.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:29:26.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:29:26.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:29:26.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:29:26.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:29:26.391 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:29:26.391 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:29:26.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:29:26.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:29:26.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:29:26.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:29:26.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:29:26.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:29:26.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:29:26.481 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:29:26.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:29:26.746 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T06:29:26.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:29:26.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:29:26.747 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:29:26.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=111669149703 2021-07-26T06:29:26.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 111669149703 2021-07-26T06:29:26.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149703' 2021-07-26T06:29:26.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:29:26.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:29:27.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=133143986178 2021-07-26T06:29:27.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 133143986178 2021-07-26T06:29:27.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149703 1-133143986178' 2021-07-26T06:29:27.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:29:27.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-111669149703 2021-07-26T06:29:27.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:29:27.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:29:27.057 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-111669149703 2021-07-26T06:29:27.057 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:29:27.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=111669149703 2021-07-26T06:29:27.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 111669149703' 2021-07-26T06:29:27.058 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 111669149703 2021-07-26T06:29:27.059 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:29:27.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149702 -lt 111669149703 2021-07-26T06:29:27.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:29:28.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:29:28.324 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:29:28.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149703 -lt 111669149703 2021-07-26T06:29:28.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:29:28.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-133143986178 2021-07-26T06:29:28.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:29:28.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:29:28.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-133143986178 2021-07-26T06:29:28.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:29:28.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=133143986178 2021-07-26T06:29:28.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 133143986178' 2021-07-26T06:29:28.600 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 133143986178 2021-07-26T06:29:28.601 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:29:28.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 133143986178 -lt 133143986178 2021-07-26T06:29:28.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:29:28.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:29:28.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:29:29.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:29:29.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:29:29.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:29:29.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:29:29.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:29:29.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:29:29.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:29:29.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:29:29.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:29:29.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:29:29.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:29:29.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:29:29.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:29:29.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:29:29.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:29:29.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:29:36.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:29:36.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:29:36.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:29:36.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:29:36.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:29:36.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:29:36.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:29:36.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:29:36.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:29:36.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:29:37.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:29:37.076 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:29:37.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:29:37.077 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:29:37.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:29:37.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:29:37.078 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:29:37.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:37.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:29:37.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:29:37.081 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:37.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:29:37.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:29:37.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:29:37.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:29:37.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:37.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:29:37.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:29:37.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:29:37.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:29:37.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:29:37.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:29:37.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:29:37.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:29:37.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:29:37.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:29:37.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:29:37.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:29:37.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:29:37.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:29:37.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:29:37.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:29:37.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:29:37.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:29:37.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:29:37.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:29:37.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:29:37.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:29:37.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:29:37.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:37.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:29:37.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:29:37.100 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:37.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:29:37.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:29:37.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:29:37.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:29:37.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:37.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:29:37.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:29:37.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:29:37.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:29:37.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:29:37.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:29:37.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:37.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:29:37.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:29:37.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:29:37.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:29:37.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:29:37.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:29:37.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2021-07-26T06:29:37.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2021-07-26T06:29:37.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local poolname=testpool 2021-07-26T06:29:37.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:442: TEST_auto_repair_bluestore_basic: run_mon td/osd-scrub-repair a 2021-07-26T06:29:37.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:29:37.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:29:37.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:29:37.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:29:37.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:29:37.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T06:29:37.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:29:37.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:37.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:37.485 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:37.485 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:37.485 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:29:37.485 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:29:37.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:29:37.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:29:37.745 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:29:37.745 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:29:37.746 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:29:37.746 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:29:37.746 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:29:37.746 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:29:37.747 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:29:37.747 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:29:37.747 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:29:37.747 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:37.748 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:29:37.748 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:29:37.748 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:29:37.749 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:29:37.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:29:37.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:29:37.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:29:37.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:29:37.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:29:37.854 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:29:37.854 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:29:37.855 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:29:37.855 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:29:37.855 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:37.855 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:29:37.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:29:37.856 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:29:37.856 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:29:37.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:443: TEST_auto_repair_bluestore_basic: run_mgr td/osd-scrub-repair x 2021-07-26T06:29:37.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:29:37.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:29:37.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:29:37.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:29:37.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:29:37.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:29:38.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:29:38.796 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:38.796 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:38.796 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:38.797 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:38.797 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:29:38.797 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:29:38.797 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:29:38.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:29:38.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:29:38.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: seq 0 2 2021-07-26T06:29:38.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: for id in $(seq 0 2) 2021-07-26T06:29:38.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:29:38.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:29:38.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:29:38.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:29:38.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:29:38.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:29:38.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:29:38.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:29:38.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:29:38.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:29:38.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:29:38.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:29:38.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:29:38.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:29:38.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:29:38.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:29:38.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:29:38.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:38.831 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:38.831 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:38.831 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:38.831 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:29:38.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:29:38.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:29:38.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:29:38.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:29:38.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:29:38.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:29:38.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:29:38.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:29:38.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:29:38.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:29:38.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:29:38.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:29:38.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:29:38.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:29:38.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8f53f9cf-70bd-4dcb-9afa-447834819136 2021-07-26T06:29:38.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8f53f9cf-70bd-4dcb-9afa-447834819136' 2021-07-26T06:29:38.836 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 8f53f9cf-70bd-4dcb-9afa-447834819136 2021-07-26T06:29:38.836 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:29:38.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSVv5g+I0GMhAAG14/FtGlVqFP9cykR4RofA== 2021-07-26T06:29:38.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSVv5g+I0GMhAAG14/FtGlVqFP9cykR4RofA=="}' 2021-07-26T06:29:38.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8f53f9cf-70bd-4dcb-9afa-447834819136 -i td/osd-scrub-repair/0/new.json 2021-07-26T06:29:39.168 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:29:39.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:29:39.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBSVv5g+I0GMhAAG14/FtGlVqFP9cykR4RofA== --osd-uuid 8f53f9cf-70bd-4dcb-9afa-447834819136 2021-07-26T06:29:39.205 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:39.206+0000 7fc08f11e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:39.207 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:39.208+0000 7fc08f11e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:39.208 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:39.209+0000 7fc08f11e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:39.208 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:39.209+0000 7fc08f11e0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:29:39.209 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:39.209+0000 7fc08f11e0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:29:39.209 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:39.209+0000 7fc08f11e0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:29:39.293 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:39.294+0000 7fc08f11e0c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T06:29:45.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:29:45.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:29:45.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:29:45.135 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:29:45.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:29:45.474 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:29:45.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:29:45.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:29:45.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:29:45.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:29:45.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:29:45.498 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:45.499+0000 7f5d5b6760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:45.500 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:45.501+0000 7f5d5b6760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:45.501 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:45.502+0000 7f5d5b6760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:45.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:29:45.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:29:45.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:29:45.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:29:45.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:29:45.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:45.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:29:45.786 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:29:45.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:45.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:46.017 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:46.017+0000 7f5d5b6760c0 -1 Falling back to public interface 2021-07-26T06:29:46.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:47.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:47.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:47.052 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:29:47.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:29:47.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:47.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:47.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:48.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:48.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:48.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:29:48.331 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:29:48.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:48.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:48.418 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:48.419+0000 7f5d5b6760c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:29:48.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:49.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:49.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:49.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:29:49.626 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:29:49.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:49.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:49.911 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/136552,v1:127.0.0.1:6803/136552] [v2:127.0.0.1:6804/136552,v1:127.0.0.1:6805/136552] exists,up 8f53f9cf-70bd-4dcb-9afa-447834819136 2021-07-26T06:29:49.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:29:49.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:29:49.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:29:49.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: for id in $(seq 0 2) 2021-07-26T06:29:49.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:29:49.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:29:49.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:29:49.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:29:49.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:29:49.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:29:49.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:29:49.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:29:49.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:29:49.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:29:49.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:29:49.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:29:49.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:29:49.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:29:49.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:29:49.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:29:49.920 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:29:49.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:49.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:49.921 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:49.921 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:49.922 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:29:49.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:29:49.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:29:49.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:29:49.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:29:49.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:29:49.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:29:49.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:29:49.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:29:49.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:29:49.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:29:49.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:29:49.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:29:49.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:29:49.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:29:49.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c11dddba-4a4f-49c4-a901-d106c10939d8 2021-07-26T06:29:49.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c11dddba-4a4f-49c4-a901-d106c10939d8' 2021-07-26T06:29:49.925 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 c11dddba-4a4f-49c4-a901-d106c10939d8 2021-07-26T06:29:49.926 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:29:49.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBdVv5gZJJ5NxAAPMd/hi4X5I7hV8oNbSRzdw== 2021-07-26T06:29:49.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBdVv5gZJJ5NxAAPMd/hi4X5I7hV8oNbSRzdw=="}' 2021-07-26T06:29:49.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c11dddba-4a4f-49c4-a901-d106c10939d8 -i td/osd-scrub-repair/1/new.json 2021-07-26T06:29:50.287 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:29:50.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:29:50.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBdVv5gZJJ5NxAAPMd/hi4X5I7hV8oNbSRzdw== --osd-uuid c11dddba-4a4f-49c4-a901-d106c10939d8 2021-07-26T06:29:50.325 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:50.326+0000 7eff9ffff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:50.326 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:50.328+0000 7eff9ffff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:50.327 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:50.329+0000 7eff9ffff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:50.328 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:50.329+0000 7eff9ffff0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:29:50.328 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:50.329+0000 7eff9ffff0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:29:50.328 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:50.329+0000 7eff9ffff0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:29:50.387 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:50.388+0000 7eff9ffff0c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T06:29:56.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:29:56.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:29:56.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:29:56.381 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:29:56.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:29:56.719 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:29:56.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:29:56.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:29:56.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:29:56.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:29:56.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:29:56.743 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:56.744+0000 7f0d00f970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:56.745 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:56.746+0000 7f0d00f970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:56.746 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:56.747+0000 7f0d00f970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:56.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:29:56.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:29:56.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:29:56.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:29:56.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:29:56.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:56.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:29:56.994 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:29:56.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:56.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:29:57.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:57.269 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:57.271+0000 7f0d00f970c0 -1 Falling back to public interface 2021-07-26T06:29:58.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:58.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:58.260 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:29:58.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:29:58.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:58.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:29:58.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:59.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:59.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:59.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:29:59.533 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:29:59.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:59.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:29:59.710 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:29:59.711+0000 7f0d00f970c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:29:59.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:00.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:00.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:00.818 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:30:00.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:30:00.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:00.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:30:01.090 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/137951,v1:127.0.0.1:6811/137951] [v2:127.0.0.1:6812/137951,v1:127.0.0.1:6813/137951] exists,up c11dddba-4a4f-49c4-a901-d106c10939d8 2021-07-26T06:30:01.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:30:01.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:30:01.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:30:01.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: for id in $(seq 0 2) 2021-07-26T06:30:01.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:30:01.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:30:01.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:30:01.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:30:01.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:30:01.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:30:01.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:30:01.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:30:01.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:30:01.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:30:01.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:30:01.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:30:01.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:30:01.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:30:01.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:30:01.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:30:01.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:30:01.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:30:01.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:30:01.101 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:30:01.102 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:01.102 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:30:01.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:30:01.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:30:01.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:30:01.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:30:01.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:30:01.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:30:01.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:30:01.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:30:01.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:30:01.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:30:01.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:30:01.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:30:01.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:30:01.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:30:01.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e835581-0cf2-4341-ab5f-ee8a918e1de3 2021-07-26T06:30:01.107 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 9e835581-0cf2-4341-ab5f-ee8a918e1de3 2021-07-26T06:30:01.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9e835581-0cf2-4341-ab5f-ee8a918e1de3' 2021-07-26T06:30:01.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:30:01.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpVv5gN+aBBhAAXBmwLHBa0UIbqX4Y29/CKg== 2021-07-26T06:30:01.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpVv5gN+aBBhAAXBmwLHBa0UIbqX4Y29/CKg=="}' 2021-07-26T06:30:01.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e835581-0cf2-4341-ab5f-ee8a918e1de3 -i td/osd-scrub-repair/2/new.json 2021-07-26T06:30:01.496 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:30:01.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2021-07-26T06:30:01.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBpVv5gN+aBBhAAXBmwLHBa0UIbqX4Y29/CKg== --osd-uuid 9e835581-0cf2-4341-ab5f-ee8a918e1de3 2021-07-26T06:30:01.534 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:01.535+0000 7f4c0e6750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:01.554 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:01.555+0000 7f4c0e6750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:01.555 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:01.556+0000 7f4c0e6750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:01.555 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:01.556+0000 7f4c0e6750c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:30:01.555 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:01.556+0000 7f4c0e6750c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:30:01.556 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:01.556+0000 7f4c0e6750c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:30:01.654 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:01.655+0000 7f4c0e6750c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2021-07-26T06:30:07.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T06:30:07.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:30:07.671 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:30:07.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:30:07.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:30:08.016 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:30:08.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:30:08.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:30:08.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:30:08.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:30:08.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:30:08.040 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:08.041+0000 7fedb45c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:08.063 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:08.061+0000 7fedb45c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:08.064 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:08.062+0000 7fedb45c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:08.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:30:08.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:30:08.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:30:08.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:30:08.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:30:08.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:08.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:30:08.307 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:30:08.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:08.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:30:08.578 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:08.579+0000 7fedb45c50c0 -1 Falling back to public interface 2021-07-26T06:30:08.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:09.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:09.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:09.606 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:30:09.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:30:09.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:09.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:30:09.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:10.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:10.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:10.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:30:10.886 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:30:10.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:10.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:30:11.037 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:11.038+0000 7fedb45c50c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:30:11.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:12.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:12.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:12.246 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:30:12.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:30:12.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:12.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:30:12.522 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/139354,v1:127.0.0.1:6819/139354] [v2:127.0.0.1:6820/139354,v1:127.0.0.1:6821/139354] exists,up 9e835581-0cf2-4341-ab5f-ee8a918e1de3 2021-07-26T06:30:12.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:30:12.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:30:12.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:30:12.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: create_pool testpool 1 1 2021-07-26T06:30:12.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2021-07-26T06:30:12.990 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'testpool' created 2021-07-26T06:30:12.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:30:14.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:452: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2021-07-26T06:30:14.643 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 1 size to 2 2021-07-26T06:30:14.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:453: TEST_auto_repair_bluestore_basic: wait_for_clean 2021-07-26T06:30:14.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:30:14.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:30:14.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:30:14.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:30:14.655 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:30:14.655 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:30:14.656 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:30:14.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:30:14.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:30:14.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:30:14.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:30:14.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:30:14.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:30:14.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:30:14.749 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:30:15.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:30:15.120 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:30:15.121 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:30:15.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:30:15.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:30:15.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:30:15.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836487 2021-07-26T06:30:15.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836487 2021-07-26T06:30:15.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487' 2021-07-26T06:30:15.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:30:15.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:30:15.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672964 2021-07-26T06:30:15.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672964 2021-07-26T06:30:15.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2021-07-26T06:30:15.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:30:15.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:30:15.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509442 2021-07-26T06:30:15.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509442 2021-07-26T06:30:15.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2021-07-26T06:30:15.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:30:15.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836487 2021-07-26T06:30:15.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:30:15.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:30:15.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836487 2021-07-26T06:30:15.558 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:30:15.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836487 2021-07-26T06:30:15.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-07-26T06:30:15.559 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836487 2021-07-26T06:30:15.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:30:15.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836487 2021-07-26T06:30:15.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:30:15.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672964 2021-07-26T06:30:15.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:30:15.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:30:15.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672964 2021-07-26T06:30:15.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:30:15.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672964 2021-07-26T06:30:15.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2021-07-26T06:30:15.831 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672964 2021-07-26T06:30:15.831 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:30:16.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672963 -lt 42949672964 2021-07-26T06:30:16.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:30:17.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:30:17.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:30:17.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672963 -lt 42949672964 2021-07-26T06:30:17.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:30:18.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:30:18.416 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:30:18.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672965 -lt 42949672964 2021-07-26T06:30:18.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:30:18.692 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509442 2021-07-26T06:30:18.692 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:30:18.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:30:18.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509442 2021-07-26T06:30:18.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:30:18.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509442 2021-07-26T06:30:18.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2021-07-26T06:30:18.695 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509442 2021-07-26T06:30:18.695 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:30:18.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509442 -lt 64424509442 2021-07-26T06:30:18.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:30:18.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:30:18.963 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:30:19.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:30:19.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:30:19.285 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:30:19.285 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:30:19.286 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:30:19.286 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:30:19.286 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:30:19.287 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:30:19.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:30:19.548 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:30:19.548 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:30:19.548 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:30:19.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:30:19.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:30:19.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:30:19.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2021-07-26T06:30:19.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:457: TEST_auto_repair_bluestore_basic: echo ABCDEF 2021-07-26T06:30:19.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:458: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T06:30:20.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:462: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2021-07-26T06:30:20.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T06:30:20.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:30:20.038 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool SOMETHING 2021-07-26T06:30:20.038 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:30:20.038 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:30:20.038 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:30:20.039 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:30:20.306 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:30:20.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:30:20.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:30:20.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:462: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:30:20.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:30:20.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:30:20.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:30:20.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:30:20.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:30:20.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:30:20.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:30:20.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:30:20.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:30:20.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:30:20.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:30:20.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:30:20.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:30:20.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:30:20.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:30:20.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:30:20.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:30:20.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:30:20.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:30:20.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:30:20.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:30:20.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:30:20.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2021-07-26T06:30:22.440 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:30:22.441 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:eb822e21:::SOMETHING:head# 2021-07-26T06:30:22.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:30:22.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:30:22.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:30:22.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:30:22.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:30:22.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:30:22.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:30:22.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:30:22.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:30:22.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:30:22.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:30:22.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:30:22.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:30:22.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:30:22.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:30:22.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:30:22.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:30:22.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:30:22.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:30:22.970 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:30:22.970 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:22.971 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:30:22.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:30:22.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:30:22.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:30:22.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:30:22.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:30:22.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:30:22.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:30:22.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:30:22.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:30:22.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:30:22.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:30:22.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:30:22.974 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:30:22.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:30:22.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:30:22.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:30:22.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:30:22.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:30:22.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:30:22.991 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:22.992+0000 7f493c5b90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:22.993 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:22.994+0000 7f493c5b90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:22.994 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:22.995+0000 7f493c5b90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:23.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:30:23.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:30:23.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:30:23.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:30:23.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:30:23.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:23.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:30:23.247 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:30:23.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:23.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:23.517 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:23.518+0000 7f493c5b90c0 -1 Falling back to public interface 2021-07-26T06:30:23.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:24.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:24.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:24.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:30:24.533 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:30:24.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:24.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:24.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:25.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:25.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:25.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:30:25.807 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:30:25.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:25.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:26.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:26.145 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:26.145+0000 7f493c5b90c0 -1 osd.0 21 log_to_monitors {default=true} 2021-07-26T06:30:27.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:27.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:27.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:30:27.103 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:30:27.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:27.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:27.393 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/140853,v1:127.0.0.1:6803/140853] [v2:127.0.0.1:6804/140853,v1:127.0.0.1:6805/140853] exists,up 8f53f9cf-70bd-4dcb-9afa-447834819136 2021-07-26T06:30:27.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:30:27.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:30:27.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:30:27.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:30:27.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:30:27.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:30:27.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:30:27.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:30:27.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:30:27.396 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:30:27.396 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:30:27.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:30:27.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:30:27.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:30:27.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:30:27.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:30:27.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:30:27.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:30:27.488 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:30:27.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:30:27.939 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:30:27.939 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:30:27.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:30:27.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:30:27.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:30:28.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182402 2021-07-26T06:30:28.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182402 2021-07-26T06:30:28.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182402' 2021-07-26T06:30:28.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:30:28.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:30:28.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672968 2021-07-26T06:30:28.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672968 2021-07-26T06:30:28.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182402 1-42949672968' 2021-07-26T06:30:28.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:30:28.217 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:30:28.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509446 2021-07-26T06:30:28.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509446 2021-07-26T06:30:28.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182402 1-42949672968 2-64424509446' 2021-07-26T06:30:28.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:30:28.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-107374182402 2021-07-26T06:30:28.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:30:28.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:30:28.337 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-107374182402 2021-07-26T06:30:28.337 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:30:28.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182402 2021-07-26T06:30:28.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2021-07-26T06:30:28.338 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 107374182402 2021-07-26T06:30:28.339 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:30:28.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 107374182402 2021-07-26T06:30:28.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:30:29.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:30:29.603 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:30:29.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182402 -lt 107374182402 2021-07-26T06:30:29.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:30:29.880 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672968 2021-07-26T06:30:29.880 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:30:29.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:30:29.881 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672968 2021-07-26T06:30:29.881 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:30:29.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672968 2021-07-26T06:30:29.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-07-26T06:30:29.883 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672968 2021-07-26T06:30:29.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:30:30.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672968 -lt 42949672968 2021-07-26T06:30:30.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:30:30.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509446 2021-07-26T06:30:30.178 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:30:30.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:30:30.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509446 2021-07-26T06:30:30.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:30:30.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509446 2021-07-26T06:30:30.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-07-26T06:30:30.180 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509446 2021-07-26T06:30:30.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:30:30.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509446 -lt 64424509446 2021-07-26T06:30:30.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:30:30.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:30:30.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:30:30.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:30:30.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:30:30.814 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:30:30.815 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:30:30.815 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:30:30.815 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:30:30.815 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:30:30.815 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:30:31.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:30:31.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:30:31.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:30:31.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:30:31.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:30:31.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:30:31.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:30:31.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2021-07-26T06:30:31.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=testpool 2021-07-26T06:30:31.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:30:31.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map testpool SOMETHING 2021-07-26T06:30:31.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:30:31.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local pgid=1.0 2021-07-26T06:30:31.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2021-07-26T06:30:31.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:30:31.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:30:31.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:30:31.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:30:32.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local primary=1 2021-07-26T06:30:32.069 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:466: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2021-07-26T06:30:32.069 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:32.070 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:32.070 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:32.070 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:32.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:466: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:32.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:467: TEST_auto_repair_bluestore_basic: ceph tell 1.0 deep_scrub 2021-07-26T06:30:32.477 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:30:32.478 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": true, 2021-07-26T06:30:32.478 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:28:52.479468+0000" 2021-07-26T06:30:32.478 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:30:32.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 scrub 2021-07-26T06:30:32.599 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:30:32.599 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:30:32.599 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:28:52.600703+0000" 2021-07-26T06:30:32.600 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:30:32.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:32.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.0 2021-07-26T06:30:32.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:32.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:30:32.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:30:32.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:30:32.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:30:32.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:32.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:32.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:32.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:32.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:30:12.877561+0000 '>' 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:32.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:30:33.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:30:33.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:30:33.873 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:30:33.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:33.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:33.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:33.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:34.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:30:12.877561+0000 '>' 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:34.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:30:35.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:30:35.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:30:35.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:30:35.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:35.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:35.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:35.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:35.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:30:12.877561+0000 '>' 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:35.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:30:36.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:30:36.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:30:36.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:30:36.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:36.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:36.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:36.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:36.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:30:12.877561+0000 '>' 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:36.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:30:37.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:30:37.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:30:37.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:30:37.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:37.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:37.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:37.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:37.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T06:28:52.600703+0000 '>' 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:37.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:30:38.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:30:38.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:30:38.999 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:30:39.000 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:39.000 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:39.000 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:39.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:39.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T06:28:52.600703+0000 '>' 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:39.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:30:40.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:30:40.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:30:40.272 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:30:40.272 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:40.272 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:40.273 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:40.273 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:40.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T06:28:52.600703+0000 '>' 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:40.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:30:41.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:30:41.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:30:41.544 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:30:41.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:41.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:41.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:41.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:41.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T06:28:52.600703+0000 '>' 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:41.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:30:42.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:30:42.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:30:42.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:30:42.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:42.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:42.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:42.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:43.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T06:28:52.600703+0000 '>' 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:43.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:30:44.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:30:44.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:30:44.091 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:30:44.091 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:44.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:44.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:44.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:44.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T06:28:52.600703+0000 '>' 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:44.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:30:45.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:30:45.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:30:45.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:30:45.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:45.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:45.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:45.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:45.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T06:28:52.600703+0000 '>' 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:45.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:30:46.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:30:46.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:30:46.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:30:46.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:46.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:46.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:46.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:46.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T06:28:52.600703+0000 '>' 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:46.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:30:47.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:30:47.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:30:47.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:30:47.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:30:47.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:30:47.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:30:47.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:30:48.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:30:41.407229+0000 '>' 2021-07-26T06:30:12.877561+0000 2021-07-26T06:30:48.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:30:48.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2021-07-26T06:30:48.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:30:48.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:30:48.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:30:48.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:30:48.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:30:48.187 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:30:48.187 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:30:48.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:30:48.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:30:48.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:30:48.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:30:48.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:30:48.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:30:48.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:30:48.280 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:30:48.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:30:48.545 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:30:48.546 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:30:48.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:30:48.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:30:48.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:30:48.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182407 2021-07-26T06:30:48.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182407 2021-07-26T06:30:48.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182407' 2021-07-26T06:30:48.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:30:48.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:30:48.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672973 2021-07-26T06:30:48.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672973 2021-07-26T06:30:48.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182407 1-42949672973' 2021-07-26T06:30:48.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:30:48.784 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:30:48.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509451 2021-07-26T06:30:48.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509451 2021-07-26T06:30:48.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182407 1-42949672973 2-64424509451' 2021-07-26T06:30:48.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:30:48.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-107374182407 2021-07-26T06:30:48.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:30:48.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:30:48.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-107374182407 2021-07-26T06:30:48.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:30:48.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182407 2021-07-26T06:30:48.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2021-07-26T06:30:48.905 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 107374182407 2021-07-26T06:30:48.905 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:30:49.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182406 -lt 107374182407 2021-07-26T06:30:49.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:30:50.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:30:50.171 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:30:50.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182407 -lt 107374182407 2021-07-26T06:30:50.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:30:50.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672973 2021-07-26T06:30:50.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:30:50.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:30:50.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672973 2021-07-26T06:30:50.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:30:50.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672973 2021-07-26T06:30:50.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2021-07-26T06:30:50.450 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672973 2021-07-26T06:30:50.450 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:30:50.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672973 -lt 42949672973 2021-07-26T06:30:50.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:30:50.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509451 2021-07-26T06:30:50.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:30:50.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:30:50.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509451 2021-07-26T06:30:50.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:30:50.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509451 2021-07-26T06:30:50.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2021-07-26T06:30:50.720 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509451 2021-07-26T06:30:50.720 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:30:50.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509451 -lt 64424509451 2021-07-26T06:30:50.985 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:30:50.986 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:30:50.986 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:30:51.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:30:51.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:30:51.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:30:51.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:30:51.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:30:51.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:30:51.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:30:51.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:30:51.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:30:51.572 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:30:51.572 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:30:51.572 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:30:51.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:30:51.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:30:51.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:30:51.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2021-07-26T06:30:52.176 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:30:52.177 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 1 0 0 0 0 7 0 0 1 1 active+clean 2021-07-26T06:30:41.748979+0000 21'1 26:53 [1,0] 1 [1,0] 1 21'1 2021-07-26T06:30:41.407229+0000 21'1 2021-07-26T06:30:41.407229+0000 0 2021-07-26T06:30:52.177 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:30:52.177 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:30:52.178 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:30:52.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2021-07-26T06:30:52.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T06:30:52.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:30:52.190 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool SOMETHING 2021-07-26T06:30:52.190 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:30:52.190 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:30:52.190 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:30:52.191 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:30:52.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:30:52.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:30:52.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:30:52.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2021-07-26T06:30:52.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:30:52.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:30:52.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:30:52.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:30:52.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2021-07-26T06:30:52.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:30:52.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:30:52.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:30:52.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:30:52.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:30:52.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:30:52.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:30:52.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:30:52.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:30:52.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:30:52.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:30:52.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2021-07-26T06:30:52.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:30:52.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:30:52.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:30:52.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:30:52.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:30:52.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2021-07-26T06:30:54.325 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:30:54.326 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:30:54.326 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:30:54.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:30:54.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:30:54.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:30:54.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:30:54.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:30:54.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:30:54.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:30:54.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:30:54.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:30:54.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:30:54.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:30:54.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:30:54.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:30:54.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:30:54.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:30:54.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:30:54.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:30:54.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:30:54.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:30:54.606 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:30:54.607 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:54.607 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:30:54.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:30:54.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:30:54.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:30:54.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:30:54.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:30:54.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:30:54.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:30:54.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:30:54.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:30:54.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:30:54.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:30:54.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:30:54.610 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:30:54.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:30:54.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:30:54.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:30:54.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:30:54.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:30:54.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:30:54.628 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:54.629+0000 7f2dabd3f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:54.629 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:54.631+0000 7f2dabd3f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:54.630 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:54.632+0000 7f2dabd3f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:54.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:30:54.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:30:54.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:30:54.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:30:54.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:30:54.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:54.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:30:54.882 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:30:54.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:54.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:55.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:55.154 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:55.155+0000 7f2dabd3f0c0 -1 Falling back to public interface 2021-07-26T06:30:56.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:56.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:56.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:30:56.152 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:30:56.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:56.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:56.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:57.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:57.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:57.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:30:57.428 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:30:57.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:57.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:57.618 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:30:57.619+0000 7f2dabd3f0c0 -1 osd.0 26 log_to_monitors {default=true} 2021-07-26T06:30:57.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:58.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:58.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:58.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:30:58.787 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:30:58.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:58.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:59.083 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/143114,v1:127.0.0.1:6803/143114] [v2:127.0.0.1:6804/143114,v1:127.0.0.1:6805/143114] exists,up 8f53f9cf-70bd-4dcb-9afa-447834819136 2021-07-26T06:30:59.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:30:59.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:30:59.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:30:59.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:30:59.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:30:59.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:30:59.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:30:59.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:30:59.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:30:59.086 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:30:59.086 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:30:59.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:30:59.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:30:59.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:30:59.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:30:59.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:30:59.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:30:59.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:30:59.178 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:30:59.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:30:59.443 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:30:59.444 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:30:59.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:30:59.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:30:59.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:30:59.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018882 2021-07-26T06:30:59.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018882 2021-07-26T06:30:59.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018882' 2021-07-26T06:30:59.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:30:59.563 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:30:59.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672976 2021-07-26T06:30:59.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672976 2021-07-26T06:30:59.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018882 1-42949672976' 2021-07-26T06:30:59.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:30:59.682 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:30:59.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509454 2021-07-26T06:30:59.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509454 2021-07-26T06:30:59.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018882 1-42949672976 2-64424509454' 2021-07-26T06:30:59.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:30:59.800 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-128849018882 2021-07-26T06:30:59.800 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:30:59.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:30:59.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-128849018882 2021-07-26T06:30:59.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:30:59.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018882 2021-07-26T06:30:59.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2021-07-26T06:30:59.803 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 128849018882 2021-07-26T06:30:59.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:31:00.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 128849018882 2021-07-26T06:31:00.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:31:01.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:31:01.116 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:31:01.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 128849018882 2021-07-26T06:31:01.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:31:02.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:31:02.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:31:02.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018882 -lt 128849018882 2021-07-26T06:31:02.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:02.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672976 2021-07-26T06:31:02.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:02.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:31:02.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672976 2021-07-26T06:31:02.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:02.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672976 2021-07-26T06:31:02.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2021-07-26T06:31:02.672 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672976 2021-07-26T06:31:02.672 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:31:02.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672977 -lt 42949672976 2021-07-26T06:31:02.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:02.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509454 2021-07-26T06:31:02.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:02.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:31:02.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509454 2021-07-26T06:31:02.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:02.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509454 2021-07-26T06:31:02.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2021-07-26T06:31:02.943 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509454 2021-07-26T06:31:02.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:31:03.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509454 -lt 64424509454 2021-07-26T06:31:03.210 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:31:03.210 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:31:03.210 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:31:03.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:31:03.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:31:03.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:31:03.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:31:03.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:31:03.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:31:03.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:31:03.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:31:03.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:31:03.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:31:03.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:31:03.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:31:04.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:31:04.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:31:04.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:31:04.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2021-07-26T06:31:04.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T06:31:04.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:31:04.159 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool SOMETHING 2021-07-26T06:31:04.159 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:31:04.159 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:31:04.159 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:31:04.159 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:31:04.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:31:04.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:31:04.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:31:04.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2021-07-26T06:31:04.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:31:04.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:31:04.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:31:04.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:31:04.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2021-07-26T06:31:04.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:31:04.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:31:04.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:31:04.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:31:04.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:31:04.737 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:31:04.737 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:31:04.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:31:04.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:31:04.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:31:04.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:31:04.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2021-07-26T06:31:04.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:31:04.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:31:04.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:31:04.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:31:04.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:31:04.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2021-07-26T06:31:06.514 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:31:07.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:31:07.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:31:07.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:31:07.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:31:07.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:31:07.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:31:07.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:31:07.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:31:07.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:31:07.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:31:07.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:31:07.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:31:07.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:31:07.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:31:07.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:31:07.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:31:07.058 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:31:07.058 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:31:07.058 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:31:07.058 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:31:07.059 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:07.059 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:31:07.059 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:31:07.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:31:07.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:31:07.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:31:07.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:31:07.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:31:07.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:31:07.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:31:07.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:31:07.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:31:07.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:31:07.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:31:07.062 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:31:07.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:31:07.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:31:07.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:31:07.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:31:07.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:31:07.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:31:07.079 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:07.079+0000 7f98ec0a00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:07.080 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:07.081+0000 7f98ec0a00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:07.081 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:07.082+0000 7f98ec0a00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:07.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:31:07.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:31:07.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:31:07.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:31:07.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:31:07.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:07.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:31:07.331 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:31:07.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:07.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:07.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:07.612 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:07.613+0000 7f98ec0a00c0 -1 Falling back to public interface 2021-07-26T06:31:08.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:08.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:08.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:31:08.601 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:31:08.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:08.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:08.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:09.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:09.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:09.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:31:09.880 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:31:09.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:09.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:10.104 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:10.104+0000 7f98ec0a00c0 -1 osd.0 31 log_to_monitors {default=true} 2021-07-26T06:31:10.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:11.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:11.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:11.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:31:11.160 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:31:11.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:11.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:11.456 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/144536,v1:127.0.0.1:6803/144536] [v2:127.0.0.1:6804/144536,v1:127.0.0.1:6805/144536] exists,up 8f53f9cf-70bd-4dcb-9afa-447834819136 2021-07-26T06:31:11.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:31:11.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:31:11.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:31:11.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:31:11.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:31:11.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:31:11.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:31:11.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:31:11.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:31:11.458 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:31:11.459 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:31:11.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:31:11.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:31:11.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:31:11.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:31:11.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:31:11.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:31:11.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:31:11.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:31:11.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:31:11.907 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:31:11.907 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:31:11.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:31:11.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:31:11.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:31:12.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855362 2021-07-26T06:31:12.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855362 2021-07-26T06:31:12.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-150323855362' 2021-07-26T06:31:12.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:31:12.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:31:12.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672980 2021-07-26T06:31:12.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672980 2021-07-26T06:31:12.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-150323855362 1-42949672980' 2021-07-26T06:31:12.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:31:12.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:31:12.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509457 2021-07-26T06:31:12.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509457 2021-07-26T06:31:12.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-150323855362 1-42949672980 2-64424509457' 2021-07-26T06:31:12.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:12.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-150323855362 2021-07-26T06:31:12.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:12.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:31:12.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-150323855362 2021-07-26T06:31:12.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:12.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855362 2021-07-26T06:31:12.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2021-07-26T06:31:12.268 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 150323855362 2021-07-26T06:31:12.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:31:12.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 150323855362 2021-07-26T06:31:12.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:31:13.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:31:13.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:31:13.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855362 -lt 150323855362 2021-07-26T06:31:13.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:13.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672980 2021-07-26T06:31:13.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:13.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:31:13.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672980 2021-07-26T06:31:13.814 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:13.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672980 2021-07-26T06:31:13.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2021-07-26T06:31:13.815 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672980 2021-07-26T06:31:13.815 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:31:14.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672980 -lt 42949672980 2021-07-26T06:31:14.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:14.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509457 2021-07-26T06:31:14.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:14.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:31:14.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509457 2021-07-26T06:31:14.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:14.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509457 2021-07-26T06:31:14.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2021-07-26T06:31:14.120 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509457 2021-07-26T06:31:14.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:31:14.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509458 -lt 64424509457 2021-07-26T06:31:14.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:31:14.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:31:14.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:31:14.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:31:14.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:31:14.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:31:14.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:31:14.786 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:31:14.786 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:31:14.786 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:31:14.786 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:31:15.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:31:15.131 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:31:15.131 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:31:15.131 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:31:15.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:31:15.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:31:15.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:31:15.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:31:15.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2021-07-26T06:31:15.457 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:30:41.406+0000 7f0cdf8dc700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber pg(1.0) scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2021-07-26T06:31:15.457 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:30:41.406+0000 7f0cdf8dc700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] PrimaryLog scrubber pg(1.0) _scrub_finish info stats: valid 2021-07-26T06:31:15.458 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:30:41.406+0000 7f0cdf8dc700 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber pg(1.0) scrub_finish All may be fixed 2021-07-26T06:31:15.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:31:15.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:31:15.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:31:15.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:31:15.460 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:31:15.460 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:31:15.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:31:15.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:31:15.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:31:15.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:31:15.565 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:31:15.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:31:15.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:31:15.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:31:15.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:31:15.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:31:15.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:31:15.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:31:15.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:31:15.570 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:31:15.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:31:15.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:31:15.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:31:15.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:31:15.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:15.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:31:15.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:31:15.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:31:15.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:31:15.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:31:15.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:31:15.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:31:15.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:31:15.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:31:15.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:31:15.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:31:15.583 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:31:15.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:31:15.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:31:15.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:31:15.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:31:15.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:31:15.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:31:15.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:31:15.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:31:15.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:31:15.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:31:15.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:31:15.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:31:15.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:31:15.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:31:15.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:31:15.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:31:15.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:31:15.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:31:15.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:31:15.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:15.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:31:15.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:31:15.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:31:15.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:31:15.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:31:15.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:31:15.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:15.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:31:15.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:31:15.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:31:15.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:31:15.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:31:15.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:31:15.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2021-07-26T06:31:15.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2021-07-26T06:31:15.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:536: TEST_auto_repair_bluestore_failed: local poolname=testpool 2021-07-26T06:31:15.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2021-07-26T06:31:15.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:31:15.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:31:15.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:31:15.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:31:15.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:31:15.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T06:31:16.000 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:31:16.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:31:16.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:31:16.001 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:31:16.001 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:16.002 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:31:16.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:31:16.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:31:16.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:31:16.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:31:16.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:31:16.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:31:16.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:31:16.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:31:16.312 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:31:16.312 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:31:16.312 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:31:16.313 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:31:16.313 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:16.313 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:31:16.313 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:31:16.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:31:16.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:31:16.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:31:16.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:31:16.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:31:16.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:31:16.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:31:16.413 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:31:16.414 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:31:16.414 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:31:16.414 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:31:16.414 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:16.414 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:31:16.415 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:31:16.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:31:16.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:31:16.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2021-07-26T06:31:16.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:31:16.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:31:16.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:31:16.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:31:16.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:31:16.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:31:17.354 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:31:17.355 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:31:17.355 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:31:17.355 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:31:17.355 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:17.356 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:31:17.356 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:31:17.356 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:31:17.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:31:17.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:31:17.379 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: seq 0 2 2021-07-26T06:31:17.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2021-07-26T06:31:17.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:545: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:31:17.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:31:17.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:31:17.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:31:17.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:31:17.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:31:17.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:31:17.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:31:17.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:31:17.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:31:17.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:31:17.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:31:17.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:31:17.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:31:17.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:31:17.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:31:17.389 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:31:17.389 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:31:17.389 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:31:17.390 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:31:17.390 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:17.390 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:31:17.390 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:31:17.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:31:17.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:31:17.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:31:17.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:31:17.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:31:17.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:31:17.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:31:17.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:31:17.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:31:17.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:31:17.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:31:17.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:31:17.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:31:17.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9085dda3-8ee1-40d5-a9de-62ff8a8755a0 2021-07-26T06:31:17.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9085dda3-8ee1-40d5-a9de-62ff8a8755a0' 2021-07-26T06:31:17.395 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 9085dda3-8ee1-40d5-a9de-62ff8a8755a0 2021-07-26T06:31:17.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:31:17.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1Vv5gSuq+FxAA/OMGf3zOxGh1+LuHnm3j/w== 2021-07-26T06:31:17.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1Vv5gSuq+FxAA/OMGf3zOxGh1+LuHnm3j/w=="}' 2021-07-26T06:31:17.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9085dda3-8ee1-40d5-a9de-62ff8a8755a0 -i td/osd-scrub-repair/0/new.json 2021-07-26T06:31:17.718 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:31:17.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:31:17.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQC1Vv5gSuq+FxAA/OMGf3zOxGh1+LuHnm3j/w== --osd-uuid 9085dda3-8ee1-40d5-a9de-62ff8a8755a0 2021-07-26T06:31:17.755 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:17.755+0000 7fbf8ec100c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:17.757 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:17.757+0000 7fbf8ec100c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:17.758 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:17.758+0000 7fbf8ec100c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:17.758 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:17.758+0000 7fbf8ec100c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:31:17.759 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:17.758+0000 7fbf8ec100c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:31:17.759 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:17.758+0000 7fbf8ec100c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:31:17.826 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:17.827+0000 7fbf8ec100c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T06:31:23.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:31:23.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:31:23.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:31:23.571 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:31:23.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:31:23.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:31:23.916 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:31:23.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:31:23.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:31:23.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:31:23.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:31:23.939 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:23.940+0000 7f0c5676a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:23.941 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:23.941+0000 7f0c5676a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:23.942 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:23.942+0000 7f0c5676a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:24.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:31:24.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:31:24.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:31:24.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:31:24.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:31:24.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:24.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:31:24.192 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:31:24.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:24.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:24.456 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:24.457+0000 7f0c5676a0c0 -1 Falling back to public interface 2021-07-26T06:31:24.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:25.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:25.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:25.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:31:25.500 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:31:25.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:25.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:25.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:26.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:26.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:26.770 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:31:26.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:31:26.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:26.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:26.899 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:26.899+0000 7f0c5676a0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:31:27.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:28.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:28.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:28.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:31:28.074 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:31:28.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:28.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:28.351 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/146631,v1:127.0.0.1:6803/146631] [v2:127.0.0.1:6804/146631,v1:127.0.0.1:6805/146631] exists,up 9085dda3-8ee1-40d5-a9de-62ff8a8755a0 2021-07-26T06:31:28.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:31:28.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:31:28.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:31:28.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2021-07-26T06:31:28.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:545: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:31:28.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:31:28.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:31:28.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:31:28.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:31:28.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:31:28.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:31:28.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:31:28.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:31:28.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:31:28.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:31:28.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:31:28.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:31:28.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:31:28.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:31:28.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:31:28.361 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:31:28.361 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:31:28.362 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:31:28.362 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:31:28.362 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:28.362 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:31:28.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:31:28.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:31:28.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:31:28.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:31:28.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:31:28.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:31:28.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:31:28.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:31:28.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:31:28.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:31:28.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:31:28.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:31:28.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:31:28.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:31:28.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ca4f7f59-f17f-40a6-b2dd-73208a8eb1cf 2021-07-26T06:31:28.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ca4f7f59-f17f-40a6-b2dd-73208a8eb1cf' 2021-07-26T06:31:28.367 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 ca4f7f59-f17f-40a6-b2dd-73208a8eb1cf 2021-07-26T06:31:28.367 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:31:28.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDAVv5gCsMTFhAAVBFP7Qb7PY4Bf6cGaowe2w== 2021-07-26T06:31:28.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDAVv5gCsMTFhAAVBFP7Qb7PY4Bf6cGaowe2w=="}' 2021-07-26T06:31:28.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ca4f7f59-f17f-40a6-b2dd-73208a8eb1cf -i td/osd-scrub-repair/1/new.json 2021-07-26T06:31:28.751 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:31:28.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:31:28.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDAVv5gCsMTFhAAVBFP7Qb7PY4Bf6cGaowe2w== --osd-uuid ca4f7f59-f17f-40a6-b2dd-73208a8eb1cf 2021-07-26T06:31:28.789 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:28.790+0000 7f5dc31290c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:28.791 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:28.792+0000 7f5dc31290c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:28.791 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:28.793+0000 7f5dc31290c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:28.792 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:28.793+0000 7f5dc31290c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:31:28.792 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:28.793+0000 7f5dc31290c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:31:28.792 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:28.793+0000 7f5dc31290c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:31:28.855 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:28.856+0000 7f5dc31290c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T06:31:34.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:31:34.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:31:34.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:31:34.599 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:31:34.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:31:34.936 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:31:34.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:31:34.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:31:34.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:31:34.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:31:34.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:31:34.960 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:34.961+0000 7f62b7af90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:34.961 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:34.963+0000 7f62b7af90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:34.962 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:34.964+0000 7f62b7af90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:35.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:31:35.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:31:35.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:31:35.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:31:35.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:31:35.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:35.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:31:35.213 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:31:35.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:31:35.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:35.481 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:35.483+0000 7f62b7af90c0 -1 Falling back to public interface 2021-07-26T06:31:35.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:36.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:36.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:36.521 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:31:36.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:31:36.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:36.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:31:36.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:37.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:37.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:37.806 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:31:37.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:31:37.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:37.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:31:37.855 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:37.856+0000 7f62b7af90c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:31:38.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:39.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:39.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:39.083 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:31:39.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:31:39.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:39.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:31:39.394 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/148030,v1:127.0.0.1:6811/148030] [v2:127.0.0.1:6812/148030,v1:127.0.0.1:6813/148030] exists,up ca4f7f59-f17f-40a6-b2dd-73208a8eb1cf 2021-07-26T06:31:39.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:31:39.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:31:39.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:31:39.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2021-07-26T06:31:39.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:545: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:31:39.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:31:39.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:31:39.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:31:39.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:31:39.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:31:39.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:31:39.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:31:39.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:31:39.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:31:39.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:31:39.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:31:39.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:31:39.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:31:39.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:31:39.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:31:39.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:31:39.405 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:31:39.405 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:31:39.405 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:31:39.405 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:39.406 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:31:39.406 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:31:39.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:31:39.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:31:39.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:31:39.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:31:39.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:31:39.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:31:39.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:31:39.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:31:39.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:31:39.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:31:39.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:31:39.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:31:39.409 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:31:39.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7c62d24c-294e-4c52-85dd-30447f5cc73a 2021-07-26T06:31:39.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7c62d24c-294e-4c52-85dd-30447f5cc73a' 2021-07-26T06:31:39.410 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 7c62d24c-294e-4c52-85dd-30447f5cc73a 2021-07-26T06:31:39.410 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:31:39.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLVv5gwTGqGBAABTO8YNHTv7iRKaFCqkiXBQ== 2021-07-26T06:31:39.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLVv5gwTGqGBAABTO8YNHTv7iRKaFCqkiXBQ=="}' 2021-07-26T06:31:39.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7c62d24c-294e-4c52-85dd-30447f5cc73a -i td/osd-scrub-repair/2/new.json 2021-07-26T06:31:39.826 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:31:39.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2021-07-26T06:31:39.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDLVv5gwTGqGBAABTO8YNHTv7iRKaFCqkiXBQ== --osd-uuid 7c62d24c-294e-4c52-85dd-30447f5cc73a 2021-07-26T06:31:39.864 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:39.865+0000 7f67bc6740c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:39.893 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:39.894+0000 7f67bc6740c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:39.894 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:39.895+0000 7f67bc6740c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:39.895 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:39.896+0000 7f67bc6740c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:31:39.895 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:39.896+0000 7f67bc6740c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:31:39.895 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:39.896+0000 7f67bc6740c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:31:39.970 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:39.971+0000 7f67bc6740c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2021-07-26T06:31:46.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T06:31:46.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:31:46.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:31:46.276 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:31:46.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:31:46.668 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:31:46.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:31:46.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:31:46.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:31:46.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:31:46.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:31:46.691 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:46.693+0000 7fabe4ad00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:46.693 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:46.694+0000 7fabe4ad00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:46.694 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:46.695+0000 7fabe4ad00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:46.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:31:46.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:31:46.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:31:46.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:31:46.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:31:46.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:46.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:31:46.979 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:31:46.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:46.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:31:47.209 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:47.211+0000 7fabe4ad00c0 -1 Falling back to public interface 2021-07-26T06:31:47.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:48.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:48.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:48.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:31:48.290 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:31:48.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:48.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:31:48.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:49.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:49.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:49.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:31:49.574 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:31:49.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:49.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:31:49.603 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:31:49.604+0000 7fabe4ad00c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:31:49.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:50.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:50.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:50.943 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:31:50.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:31:50.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:50.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:31:51.216 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/149429,v1:127.0.0.1:6819/149429] [v2:127.0.0.1:6820/149429,v1:127.0.0.1:6821/149429] exists,up 7c62d24c-294e-4c52-85dd-30447f5cc73a 2021-07-26T06:31:51.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:31:51.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:31:51.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:31:51.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2021-07-26T06:31:51.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2021-07-26T06:31:51.764 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'testpool' created 2021-07-26T06:31:51.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:31:52.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2021-07-26T06:31:53.171 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 1 size to 2 2021-07-26T06:31:53.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:550: TEST_auto_repair_bluestore_failed: wait_for_clean 2021-07-26T06:31:53.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:31:53.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:31:53.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:31:53.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:31:53.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:31:53.183 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:31:53.183 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:31:53.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:31:53.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:31:53.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:31:53.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:31:53.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:31:53.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:31:53.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:31:53.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:31:53.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:31:53.639 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:31:53.639 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:31:53.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:31:53.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:31:53.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:31:53.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836487 2021-07-26T06:31:53.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836487 2021-07-26T06:31:53.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487' 2021-07-26T06:31:53.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:31:53.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:31:53.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672964 2021-07-26T06:31:53.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672964 2021-07-26T06:31:53.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2021-07-26T06:31:53.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:31:53.920 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:31:54.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509442 2021-07-26T06:31:54.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509442 2021-07-26T06:31:54.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2021-07-26T06:31:54.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:54.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836487 2021-07-26T06:31:54.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:54.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:31:54.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836487 2021-07-26T06:31:54.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:54.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836487 2021-07-26T06:31:54.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-07-26T06:31:54.042 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836487 2021-07-26T06:31:54.043 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:31:54.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836487 2021-07-26T06:31:54.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:54.306 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672964 2021-07-26T06:31:54.306 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:54.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:31:54.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672964 2021-07-26T06:31:54.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:54.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672964 2021-07-26T06:31:54.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2021-07-26T06:31:54.312 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672964 2021-07-26T06:31:54.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:31:54.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672964 -lt 42949672964 2021-07-26T06:31:54.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:54.660 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509442 2021-07-26T06:31:54.660 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:54.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:31:54.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509442 2021-07-26T06:31:54.662 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:54.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509442 2021-07-26T06:31:54.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2021-07-26T06:31:54.663 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509442 2021-07-26T06:31:54.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:31:54.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509441 -lt 64424509442 2021-07-26T06:31:54.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:31:55.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:31:55.929 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:31:56.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509442 -lt 64424509442 2021-07-26T06:31:56.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:31:56.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:31:56.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:31:56.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:31:56.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:31:56.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:31:56.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:31:56.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:31:56.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:31:56.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:31:56.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:31:56.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:31:56.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:31:56.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:31:56.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:31:57.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:31:57.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:31:57.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:31:57.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2021-07-26T06:31:57.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: echo ABCDEF 2021-07-26T06:31:57.128 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:555: TEST_auto_repair_bluestore_failed: seq 1 10 2021-07-26T06:31:57.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:555: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2021-07-26T06:31:57.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:31:57.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:555: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2021-07-26T06:31:57.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:31:57.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:555: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2021-07-26T06:31:57.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:31:57.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:555: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2021-07-26T06:31:57.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:31:57.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:555: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2021-07-26T06:31:57.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:31:57.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:555: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2021-07-26T06:31:57.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:31:57.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:555: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2021-07-26T06:31:57.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:31:58.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:555: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2021-07-26T06:31:58.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:31:58.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:555: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2021-07-26T06:31:58.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:31:58.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:555: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2021-07-26T06:31:58.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:31:58.402 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:562: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2021-07-26T06:31:58.402 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T06:31:58.402 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:31:58.403 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool SOMETHING 2021-07-26T06:31:58.403 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:31:58.403 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:31:58.403 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:31:58.404 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:31:58.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:31:58.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:31:58.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:31:58.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:562: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2021-07-26T06:31:58.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:31:58.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:31:58.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:31:58.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:31:58.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2021-07-26T06:31:58.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:31:58.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:31:58.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:31:58.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:31:58.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:31:58.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:31:58.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:31:58.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:31:58.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:31:58.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:31:59.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:31:59.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2021-07-26T06:31:59.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:31:59.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:31:59.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:31:59.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:31:59.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:31:59.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2021-07-26T06:32:01.044 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:32:01.045 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:ff7b1f36:::obj1:head# 2021-07-26T06:32:01.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:32:01.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:32:01.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:32:01.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:32:01.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:32:01.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:32:01.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:32:01.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:32:01.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:32:01.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:32:01.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:32:01.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:32:01.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:32:01.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:32:01.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:32:01.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:32:01.569 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:32:01.569 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:32:01.570 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:32:01.570 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:32:01.570 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:01.570 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:32:01.570 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:32:01.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:32:01.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:32:01.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:32:01.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:32:01.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:32:01.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:32:01.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:32:01.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:32:01.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:32:01.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:32:01.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:32:01.573 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:32:01.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:32:01.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:32:01.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:32:01.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:32:01.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:32:01.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:32:01.591 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:01.593+0000 7f93f04ec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:01.602 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:01.604+0000 7f93f04ec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:01.603 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:01.605+0000 7f93f04ec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:01.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:32:01.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:32:01.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:32:01.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:32:01.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:32:01.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:01.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:32:01.846 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:32:01.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:01.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:02.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:02.133 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:02.133+0000 7f93f04ec0c0 -1 Falling back to public interface 2021-07-26T06:32:03.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:03.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:03.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:32:03.120 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:32:03.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:03.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:03.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:04.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:04.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:04.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:32:04.399 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:32:04.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:04.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:04.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:04.725 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:04.725+0000 7f93f04ec0c0 -1 osd.0 21 log_to_monitors {default=true} 2021-07-26T06:32:05.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:05.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:05.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:32:05.671 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:32:05.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:05.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:05.939 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/151078,v1:127.0.0.1:6803/151078] [v2:127.0.0.1:6804/151078,v1:127.0.0.1:6805/151078] exists,up 9085dda3-8ee1-40d5-a9de-62ff8a8755a0 2021-07-26T06:32:05.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:32:05.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:32:05.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:32:05.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:32:05.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:32:05.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:32:05.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:32:05.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:32:05.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:32:05.942 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:32:05.942 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:32:05.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:32:05.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:32:05.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:32:06.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:32:06.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:32:06.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:32:06.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:32:06.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:32:06.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:32:06.529 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:32:06.530 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:32:06.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:32:06.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:06.531 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:32:06.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182402 2021-07-26T06:32:06.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182402 2021-07-26T06:32:06.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182402' 2021-07-26T06:32:06.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:06.713 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:32:06.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672968 2021-07-26T06:32:06.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672968 2021-07-26T06:32:06.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182402 1-42949672968' 2021-07-26T06:32:06.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:06.831 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:32:06.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509446 2021-07-26T06:32:06.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509446 2021-07-26T06:32:06.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182402 1-42949672968 2-64424509446' 2021-07-26T06:32:06.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:06.949 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-107374182402 2021-07-26T06:32:06.949 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:06.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:32:06.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-107374182402 2021-07-26T06:32:06.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:06.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182402 2021-07-26T06:32:06.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2021-07-26T06:32:06.951 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 107374182402 2021-07-26T06:32:06.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:32:07.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 107374182402 2021-07-26T06:32:07.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:32:08.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:32:08.216 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:32:08.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182402 -lt 107374182402 2021-07-26T06:32:08.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:08.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672968 2021-07-26T06:32:08.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:08.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:32:08.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672968 2021-07-26T06:32:08.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:08.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672968 2021-07-26T06:32:08.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-07-26T06:32:08.498 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672968 2021-07-26T06:32:08.498 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:32:08.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672968 -lt 42949672968 2021-07-26T06:32:08.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:08.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509446 2021-07-26T06:32:08.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:08.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:32:08.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509446 2021-07-26T06:32:08.822 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:08.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509446 2021-07-26T06:32:08.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-07-26T06:32:08.823 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509446 2021-07-26T06:32:08.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:32:09.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509446 -lt 64424509446 2021-07-26T06:32:09.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:32:09.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:32:09.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:32:09.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:32:09.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:32:09.509 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:32:09.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:32:09.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:32:09.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:32:09.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:32:09.512 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:32:09.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:32:09.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:32:09.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:32:09.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:32:10.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:32:10.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:32:10.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:32:10.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:564: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2021-07-26T06:32:10.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T06:32:10.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:32:10.128 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool SOMETHING 2021-07-26T06:32:10.128 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:32:10.128 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:32:10.129 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:32:10.129 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:32:10.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:32:10.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:32:10.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:32:10.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:564: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2021-07-26T06:32:10.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:32:10.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:32:10.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:32:10.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:32:10.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2021-07-26T06:32:10.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:32:10.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:32:10.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:32:10.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:32:10.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:32:10.665 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:32:10.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:32:10.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:32:10.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:32:10.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:32:10.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:32:10.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2021-07-26T06:32:10.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:32:10.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:32:10.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:32:10.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:32:10.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:32:10.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2021-07-26T06:32:12.433 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:32:12.433 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:104778fc:::obj2:head# 2021-07-26T06:32:12.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:32:12.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:32:12.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:32:12.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:32:12.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:32:12.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:32:12.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:32:12.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:32:12.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:32:12.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:32:12.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:32:12.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:32:12.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:32:12.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:32:12.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:32:12.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:32:12.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:32:12.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:32:12.965 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:32:12.965 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:32:12.965 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:12.966 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:32:12.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:32:12.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:32:12.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:32:12.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:32:12.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:32:12.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:32:12.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:32:12.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:32:12.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:32:12.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:32:12.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:32:12.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:32:12.969 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:32:12.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:32:12.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:32:12.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:32:12.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:32:12.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:32:12.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:32:12.986 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:12.986+0000 7ff955dc30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:12.988 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:12.988+0000 7ff955dc30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:12.989 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:12.989+0000 7ff955dc30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:13.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:32:13.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:32:13.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:32:13.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:32:13.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:32:13.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:13.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:32:13.241 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:32:13.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:13.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:13.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:13.515 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:13.515+0000 7ff955dc30c0 -1 Falling back to public interface 2021-07-26T06:32:14.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:14.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:14.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:32:14.507 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:32:14.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:14.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:14.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:15.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:15.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:15.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:32:15.785 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:32:15.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:15.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:15.979 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:15.979+0000 7ff955dc30c0 -1 osd.0 26 log_to_monitors {default=true} 2021-07-26T06:32:16.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:17.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:17.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:17.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:32:17.069 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:32:17.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:17.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:17.348 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/152476,v1:127.0.0.1:6803/152476] [v2:127.0.0.1:6804/152476,v1:127.0.0.1:6805/152476] exists,up 9085dda3-8ee1-40d5-a9de-62ff8a8755a0 2021-07-26T06:32:17.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:32:17.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:32:17.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:32:17.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:32:17.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:32:17.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:32:17.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:32:17.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:32:17.351 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:32:17.351 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:32:17.351 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:32:17.351 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:32:17.351 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:32:17.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:32:17.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:32:17.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:32:17.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:32:17.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:32:17.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:32:17.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:32:17.709 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:32:17.709 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:32:17.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:32:17.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:17.710 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:32:17.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018882 2021-07-26T06:32:17.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018882 2021-07-26T06:32:17.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018882' 2021-07-26T06:32:17.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:17.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:32:17.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672971 2021-07-26T06:32:17.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672971 2021-07-26T06:32:17.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018882 1-42949672971' 2021-07-26T06:32:17.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:17.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:32:18.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509449 2021-07-26T06:32:18.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509449 2021-07-26T06:32:18.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018882 1-42949672971 2-64424509449' 2021-07-26T06:32:18.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:18.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-128849018882 2021-07-26T06:32:18.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:18.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:32:18.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-128849018882 2021-07-26T06:32:18.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:18.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018882 2021-07-26T06:32:18.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2021-07-26T06:32:18.105 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 128849018882 2021-07-26T06:32:18.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:32:18.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018882 -lt 128849018882 2021-07-26T06:32:18.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:18.369 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672971 2021-07-26T06:32:18.370 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:18.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:32:18.370 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672971 2021-07-26T06:32:18.371 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:18.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672971 2021-07-26T06:32:18.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2021-07-26T06:32:18.372 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672971 2021-07-26T06:32:18.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:32:18.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672970 -lt 42949672971 2021-07-26T06:32:18.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:32:19.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:32:19.673 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:32:19.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672970 -lt 42949672971 2021-07-26T06:32:19.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:32:20.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:32:20.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:32:21.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672972 -lt 42949672971 2021-07-26T06:32:21.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:21.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509449 2021-07-26T06:32:21.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:21.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:32:21.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509449 2021-07-26T06:32:21.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:21.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509449 2021-07-26T06:32:21.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2021-07-26T06:32:21.227 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509449 2021-07-26T06:32:21.228 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:32:21.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509449 -lt 64424509449 2021-07-26T06:32:21.493 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:32:21.493 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:32:21.494 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:32:21.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:32:21.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:32:21.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:32:21.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:32:21.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:32:21.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:32:21.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:32:21.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:32:22.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:32:22.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:32:22.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:32:22.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:32:22.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:32:22.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:32:22.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:32:22.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:565: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2021-07-26T06:32:22.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:32:22.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:32:22.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:32:22.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:32:22.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:565: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2021-07-26T06:32:22.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:32:22.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:32:22.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:32:22.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:32:22.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2021-07-26T06:32:22.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:32:22.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:32:22.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:32:22.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:32:22.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:32:22.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:32:22.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:32:22.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:32:22.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:32:22.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:32:22.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:32:22.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2021-07-26T06:32:22.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:32:22.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:32:22.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:32:22.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:32:22.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:32:22.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2021-07-26T06:32:24.669 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:32:25.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:32:25.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:32:25.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:32:25.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:32:25.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:32:25.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:32:25.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:32:25.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:32:25.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:32:25.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:32:25.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:32:25.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:32:25.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:32:25.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:32:25.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:32:25.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:32:25.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:32:25.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:32:25.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:32:25.195 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:32:25.196 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:25.196 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:32:25.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:32:25.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:32:25.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:32:25.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:32:25.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:32:25.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:32:25.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:32:25.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:32:25.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:32:25.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:32:25.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:32:25.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:32:25.199 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:32:25.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:32:25.200 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:32:25.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:32:25.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:32:25.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:32:25.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:32:25.216 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:25.218+0000 7f71c2e920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:25.218 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:25.220+0000 7f71c2e920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:25.219 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:25.221+0000 7f71c2e920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:25.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:32:25.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:32:25.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:32:25.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:32:25.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:32:25.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:25.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:32:25.471 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:32:25.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:25.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:32:25.737 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:25.738+0000 7f71c2e920c0 -1 Falling back to public interface 2021-07-26T06:32:25.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:26.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:26.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:26.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:32:26.826 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:32:26.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:26.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:32:27.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:28.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:28.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:28.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:32:28.116 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:32:28.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:28.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:32:28.309 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:28.310+0000 7f71c2e920c0 -1 osd.1 31 log_to_monitors {default=true} 2021-07-26T06:32:28.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:29.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:29.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:29.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:32:29.475 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:32:29.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:29.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:32:29.757 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/153881,v1:127.0.0.1:6811/153881] [v2:127.0.0.1:6812/153881,v1:127.0.0.1:6813/153881] exists,up ca4f7f59-f17f-40a6-b2dd-73208a8eb1cf 2021-07-26T06:32:29.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:32:29.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:32:29.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:32:29.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:32:29.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:32:29.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:32:29.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:32:29.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:32:29.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:32:29.759 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:32:29.760 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:32:29.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:32:29.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:32:29.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:32:29.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:32:29.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:32:29.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:32:29.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:32:29.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:32:30.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:32:30.117 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:32:30.118 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:32:30.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:32:30.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:30.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:32:30.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018885 2021-07-26T06:32:30.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018885 2021-07-26T06:32:30.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018885' 2021-07-26T06:32:30.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:30.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:32:30.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855362 2021-07-26T06:32:30.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855362 2021-07-26T06:32:30.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2021-07-26T06:32:30.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:30.355 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:32:30.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509452 2021-07-26T06:32:30.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509452 2021-07-26T06:32:30.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509452' 2021-07-26T06:32:30.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:30.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-128849018885 2021-07-26T06:32:30.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:30.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:32:30.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-128849018885 2021-07-26T06:32:30.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:30.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018885 2021-07-26T06:32:30.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2021-07-26T06:32:30.506 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 128849018885 2021-07-26T06:32:30.507 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:32:30.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018884 -lt 128849018885 2021-07-26T06:32:30.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:32:31.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:32:31.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:32:32.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018884 -lt 128849018885 2021-07-26T06:32:32.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:32:33.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:32:33.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:32:33.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018886 -lt 128849018885 2021-07-26T06:32:33.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:33.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-150323855362 2021-07-26T06:32:33.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:33.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:32:33.332 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-150323855362 2021-07-26T06:32:33.332 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:33.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855362 2021-07-26T06:32:33.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2021-07-26T06:32:33.333 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 150323855362 2021-07-26T06:32:33.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:32:33.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855362 -lt 150323855362 2021-07-26T06:32:33.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:33.638 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509452 2021-07-26T06:32:33.638 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:33.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:32:33.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509452 2021-07-26T06:32:33.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:33.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509452 2021-07-26T06:32:33.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-07-26T06:32:33.641 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509452 2021-07-26T06:32:33.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:32:33.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509453 -lt 64424509452 2021-07-26T06:32:33.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:32:33.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:32:33.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:32:34.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:32:34.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:32:34.332 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:32:34.332 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:32:34.333 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:32:34.333 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:32:34.333 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:32:34.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:32:34.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:32:34.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:32:34.637 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:32:34.637 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:32:34.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:32:34.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:32:34.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:32:34.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:567: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2021-07-26T06:32:34.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=testpool 2021-07-26T06:32:34.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=obj1 2021-07-26T06:32:34.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map testpool obj1 2021-07-26T06:32:34.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:32:35.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:567: TEST_auto_repair_bluestore_failed: local pgid=1.0 2021-07-26T06:32:35.229 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2021-07-26T06:32:35.229 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:32:35.230 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T06:32:35.230 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool obj1 2021-07-26T06:32:35.230 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:32:35.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: local primary=1 2021-07-26T06:32:35.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2021-07-26T06:32:35.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:32:35.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:32:35.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:32:35.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:32:35.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2021-07-26T06:31:51.590829+0000 2021-07-26T06:32:35.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:570: TEST_auto_repair_bluestore_failed: ceph tell 1.0 deep_scrub 2021-07-26T06:32:35.948 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:32:35.948 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": true, 2021-07-26T06:32:35.948 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:30:55.949864+0000" 2021-07-26T06:32:35.949 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:32:35.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: ceph tell 1.0 scrub 2021-07-26T06:32:36.069 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:32:36.069 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:32:36.070 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:30:56.071324+0000" 2021-07-26T06:32:36.070 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:32:36.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2021-07-26T06:31:51.590829+0000 2021-07-26T06:32:36.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.0 2021-07-26T06:32:36.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:31:51.590829+0000 2021-07-26T06:32:36.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:32:36.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:32:36.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:32:36.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:32:36.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:32:36.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:32:36.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:32:36.083 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:32:36.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:31:51.590829+0000 '>' 2021-07-26T06:31:51.590829+0000 2021-07-26T06:32:36.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:32:37.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:32:37.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:32:37.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:32:37.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:32:37.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:32:37.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:32:37.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:32:37.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:31:51.590829+0000 '>' 2021-07-26T06:31:51.590829+0000 2021-07-26T06:32:37.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:32:38.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:32:38.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:32:38.679 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:32:38.679 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:32:38.679 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:32:38.679 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:32:38.680 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:32:38.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:31:51.590829+0000 '>' 2021-07-26T06:31:51.590829+0000 2021-07-26T06:32:38.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:32:39.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:32:39.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:32:39.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:32:39.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:32:39.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:32:39.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:32:39.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:32:40.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:32:36.570595+0000 '>' 2021-07-26T06:31:51.590829+0000 2021-07-26T06:32:40.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:32:40.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:575: TEST_auto_repair_bluestore_failed: wait_for_clean 2021-07-26T06:32:40.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:32:40.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:32:40.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:32:40.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:32:40.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:32:40.227 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:32:40.227 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:32:40.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:32:40.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:32:40.228 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:32:40.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:32:40.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:32:40.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:32:40.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:32:40.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:32:40.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:32:40.586 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:32:40.586 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:32:40.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:32:40.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:40.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:32:40.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018888 2021-07-26T06:32:40.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018888 2021-07-26T06:32:40.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018888' 2021-07-26T06:32:40.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:40.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:32:40.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855365 2021-07-26T06:32:40.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855365 2021-07-26T06:32:40.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018888 1-150323855365' 2021-07-26T06:32:40.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:40.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:32:40.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509455 2021-07-26T06:32:40.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509455 2021-07-26T06:32:40.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018888 1-150323855365 2-64424509455' 2021-07-26T06:32:40.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:40.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-128849018888 2021-07-26T06:32:40.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:40.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:32:40.982 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-128849018888 2021-07-26T06:32:40.982 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:40.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018888 2021-07-26T06:32:40.983 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 128849018888 2021-07-26T06:32:40.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 128849018888' 2021-07-26T06:32:40.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:32:41.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018887 -lt 128849018888 2021-07-26T06:32:41.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:32:42.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:32:42.250 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:32:42.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018889 -lt 128849018888 2021-07-26T06:32:42.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:42.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-150323855365 2021-07-26T06:32:42.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:42.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:32:42.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-150323855365 2021-07-26T06:32:42.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:42.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855365 2021-07-26T06:32:42.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2021-07-26T06:32:42.557 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 150323855365 2021-07-26T06:32:42.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:32:42.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855365 -lt 150323855365 2021-07-26T06:32:42.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:42.822 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509455 2021-07-26T06:32:42.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:42.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:32:42.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509455 2021-07-26T06:32:42.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:42.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509455 2021-07-26T06:32:42.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2021-07-26T06:32:42.825 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509455 2021-07-26T06:32:42.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:32:43.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509456 -lt 64424509455 2021-07-26T06:32:43.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:32:43.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:32:43.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:32:43.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:32:43.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:32:43.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:32:43.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:32:43.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:32:43.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:32:43.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:32:43.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:32:43.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:32:43.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:32:43.707 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:32:43.707 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:32:44.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:32:44.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:32:44.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:32:44.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:576: TEST_auto_repair_bluestore_failed: flush_pg_stats 2021-07-26T06:32:44.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:32:44.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:32:44.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:32:44.403 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:32:44.403 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:32:44.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:32:44.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:44.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:32:44.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018890 2021-07-26T06:32:44.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018890 2021-07-26T06:32:44.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018890' 2021-07-26T06:32:44.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:44.564 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:32:44.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855367 2021-07-26T06:32:44.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855367 2021-07-26T06:32:44.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018890 1-150323855367' 2021-07-26T06:32:44.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:44.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:32:44.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509457 2021-07-26T06:32:44.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509457 2021-07-26T06:32:44.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018890 1-150323855367 2-64424509457' 2021-07-26T06:32:44.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:44.800 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-128849018890 2021-07-26T06:32:44.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:44.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:32:44.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-128849018890 2021-07-26T06:32:44.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:44.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018890 2021-07-26T06:32:44.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2021-07-26T06:32:44.804 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 128849018890 2021-07-26T06:32:44.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:32:45.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018889 -lt 128849018890 2021-07-26T06:32:45.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:32:46.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:32:46.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:32:46.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018890 -lt 128849018890 2021-07-26T06:32:46.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:46.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-150323855367 2021-07-26T06:32:46.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:46.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:32:46.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-150323855367 2021-07-26T06:32:46.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:46.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855367 2021-07-26T06:32:46.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 150323855367' 2021-07-26T06:32:46.353 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 150323855367 2021-07-26T06:32:46.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:32:46.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855367 -lt 150323855367 2021-07-26T06:32:46.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:46.659 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509457 2021-07-26T06:32:46.659 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:46.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:32:46.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509457 2021-07-26T06:32:46.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:46.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509457 2021-07-26T06:32:46.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2021-07-26T06:32:46.662 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509457 2021-07-26T06:32:46.663 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:32:46.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509457 -lt 64424509457 2021-07-26T06:32:46.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2021-07-26T06:32:46.962 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.214+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=10.216514587s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+wait pruub 11.317797661s@ [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber pg(1.0) scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2021-07-26T06:32:46.962 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.214+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=10.216514587s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+wait pruub 11.317797661s@ [ 1.0: AUTO_REPAIR ] ] PrimaryLog scrubber pg(1.0) _scrub_finish info stats: valid 2021-07-26T06:32:46.962 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.215+0000 7f71a17d7700 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=3.358258009s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair+wait pruub 11.317797661s@ [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber pg(1.0) scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2021-07-26T06:32:46.963 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.569+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=3.121513128s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+wait pruub 11.317797661s@ [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber pg(1.0) scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2021-07-26T06:32:46.963 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.569+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=3.121513128s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+wait pruub 11.317797661s@ [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] PrimaryLog scrubber pg(1.0) _scrub_finish info stats: valid 2021-07-26T06:32:46.963 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.569+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=3.121513128s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair+wait pruub 11.317797661s@ [ 1.0: REQ_SCRUB ] ] scrubber pg(1.0) scrub_finish 3 error(s) still present after re-scrub 2021-07-26T06:32:46.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2021-07-26T06:32:46.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2021-07-26T06:32:47.271 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:32:47.271 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 10 0 0 0 0 63 0 0 10 10 active+clean+inconsistent+failed_repair 2021-07-26T06:32:39.574141+0000 21'10 36:96 [1,0] 1 [1,0] 1 21'10 2021-07-26T06:32:36.570595+0000 21'10 2021-07-26T06:32:36.570595+0000 0 2021-07-26T06:32:47.271 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:32:47.272 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:32:47.272 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:32:47.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2021-07-26T06:32:47.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2021-07-26T06:32:47.534 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:32:47.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:584: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2021-07-26T06:32:47.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T06:32:47.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=obj1 2021-07-26T06:32:47.546 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool obj1 2021-07-26T06:32:47.546 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:32:47.546 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T06:32:47.546 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool obj1 2021-07-26T06:32:47.547 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:32:47.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:32:47.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool obj1 2021-07-26T06:32:47.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:32:48.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:584: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2021-07-26T06:32:48.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:32:48.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:32:48.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:32:48.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:32:48.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2021-07-26T06:32:48.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:32:48.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:32:48.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:32:48.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:32:48.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:32:48.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:32:48.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:32:48.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:32:48.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:32:48.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:32:48.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:32:48.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2021-07-26T06:32:48.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:32:48.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:32:48.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:32:48.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:32:48.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:32:48.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2021-07-26T06:32:49.723 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:32:49.724 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:32:49.724 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:32:50.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:32:50.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:32:50.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:32:50.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:32:50.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:32:50.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:32:50.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:32:50.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:32:50.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:32:50.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:32:50.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:32:50.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:32:50.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:32:50.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:32:50.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:32:50.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:32:50.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:32:50.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:32:50.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:32:50.249 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:32:50.249 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:50.249 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:32:50.249 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:32:50.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:32:50.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:32:50.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:32:50.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:32:50.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:32:50.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:32:50.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:32:50.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:32:50.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:32:50.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:32:50.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:32:50.253 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:32:50.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:32:50.253 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:32:50.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:32:50.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:32:50.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:32:50.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:32:50.271 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:50.272+0000 7f0efb7840c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:50.272 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:50.274+0000 7f0efb7840c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:50.274 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:50.275+0000 7f0efb7840c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:50.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:32:50.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:32:50.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:32:50.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:32:50.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:32:50.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:50.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:32:50.527 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:32:50.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:50.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:50.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:50.806 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:50.808+0000 7f0efb7840c0 -1 Falling back to public interface 2021-07-26T06:32:51.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:51.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:51.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:32:51.792 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:32:51.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:51.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:52.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:53.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:53.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:53.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:32:53.101 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:32:53.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:53.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:53.330 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:32:53.331+0000 7f0efb7840c0 -1 osd.0 36 log_to_monitors {default=true} 2021-07-26T06:32:53.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:54.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:54.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:54.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:32:54.383 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:32:54.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:54.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:54.725 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/156161,v1:127.0.0.1:6803/156161] [v2:127.0.0.1:6804/156161,v1:127.0.0.1:6805/156161] exists,up 9085dda3-8ee1-40d5-a9de-62ff8a8755a0 2021-07-26T06:32:54.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:32:54.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:32:54.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:32:54.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:32:54.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:32:54.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:32:54.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:32:54.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:32:54.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:32:54.727 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:32:54.728 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:32:54.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:32:54.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:32:54.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:32:54.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:32:54.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:32:54.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:32:54.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:32:54.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:32:55.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:32:55.112 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:32:55.113 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:32:55.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:32:55.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:55.114 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:32:55.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691842 2021-07-26T06:32:55.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691842 2021-07-26T06:32:55.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-171798691842' 2021-07-26T06:32:55.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:55.324 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:32:55.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855370 2021-07-26T06:32:55.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855370 2021-07-26T06:32:55.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2021-07-26T06:32:55.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:32:55.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:32:55.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509460 2021-07-26T06:32:55.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509460 2021-07-26T06:32:55.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509460' 2021-07-26T06:32:55.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:55.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-171798691842 2021-07-26T06:32:55.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:55.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:32:55.561 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-171798691842 2021-07-26T06:32:55.561 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:55.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691842 2021-07-26T06:32:55.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2021-07-26T06:32:55.561 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 171798691842 2021-07-26T06:32:55.562 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:32:55.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 171798691842 2021-07-26T06:32:55.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:32:56.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:32:56.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:32:57.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691842 -lt 171798691842 2021-07-26T06:32:57.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:57.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-150323855370 2021-07-26T06:32:57.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:57.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:32:57.128 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-150323855370 2021-07-26T06:32:57.129 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:57.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855370 2021-07-26T06:32:57.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2021-07-26T06:32:57.130 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 150323855370 2021-07-26T06:32:57.130 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:32:57.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855370 -lt 150323855370 2021-07-26T06:32:57.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:32:57.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509460 2021-07-26T06:32:57.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:32:57.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:32:57.408 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509460 2021-07-26T06:32:57.409 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:32:57.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509460 2021-07-26T06:32:57.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2021-07-26T06:32:57.410 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509460 2021-07-26T06:32:57.410 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:32:57.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509460 -lt 64424509460 2021-07-26T06:32:57.709 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:32:57.709 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:32:57.709 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:32:58.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:32:58.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:32:58.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:32:58.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:32:58.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:32:58.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:32:58.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:32:58.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:32:58.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:32:58.356 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:32:58.357 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:32:58.357 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:32:58.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:32:58.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:32:58.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:32:58.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:585: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2021-07-26T06:32:58.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:586: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:32:58.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2021-07-26T06:32:58.737 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.214+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=10.216514587s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+wait pruub 11.317797661s@ [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber pg(1.0) scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2021-07-26T06:32:58.737 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.214+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=10.216514587s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+wait pruub 11.317797661s@ [ 1.0: AUTO_REPAIR ] ] PrimaryLog scrubber pg(1.0) _scrub_finish info stats: valid 2021-07-26T06:32:58.738 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.215+0000 7f71a17d7700 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=3.358258009s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair+wait pruub 11.317797661s@ [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber pg(1.0) scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2021-07-26T06:32:58.738 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.569+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=3.121513128s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+wait pruub 11.317797661s@ [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber pg(1.0) scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2021-07-26T06:32:58.738 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.569+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=3.121513128s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+wait pruub 11.317797661s@ [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] PrimaryLog scrubber pg(1.0) _scrub_finish info stats: valid 2021-07-26T06:32:58.738 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.569+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=3.121513128s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair+wait pruub 11.317797661s@ [ 1.0: REQ_SCRUB ] ] scrubber pg(1.0) scrub_finish 3 error(s) still present after re-scrub 2021-07-26T06:32:58.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2021-07-26T06:32:58.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:32:58.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:32:58.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:32:58.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:32:59.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2021-07-26T06:32:59.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:32:59.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:32:59.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:32:59.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:32:59.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2021-07-26T06:32:59.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:32:59.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:32:59.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:32:59.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:32:59.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:32:59.011 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:32:59.011 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:32:59.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:32:59.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:32:59.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:32:59.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:32:59.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2021-07-26T06:32:59.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:32:59.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:32:59.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:32:59.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:32:59.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:32:59.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2021-07-26T06:33:00.618 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:33:00.619 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2021-07-26T06:33:00.619 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:104778fc:::obj2:head# 2021-07-26T06:33:01.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:33:01.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:33:01.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:33:01.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:33:01.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:33:01.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:33:01.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:33:01.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:33:01.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:33:01.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:33:01.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:33:01.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:33:01.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:33:01.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:33:01.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:33:01.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:33:01.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:33:01.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:33:01.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:33:01.149 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:33:01.149 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:01.149 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:33:01.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:33:01.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:33:01.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:33:01.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:33:01.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:33:01.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:33:01.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:33:01.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:33:01.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:33:01.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:33:01.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:33:01.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:33:01.153 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:33:01.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:33:01.153 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:33:01.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:33:01.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:33:01.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:33:01.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:33:01.169 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:01.169+0000 7f6dd55790c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:01.171 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:01.171+0000 7f6dd55790c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:01.172 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:01.172+0000 7f6dd55790c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:01.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:33:01.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:33:01.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:33:01.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:33:01.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:33:01.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:01.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:33:01.422 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:33:01.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:01.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:33:01.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:01.693 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:01.694+0000 7f6dd55790c0 -1 Falling back to public interface 2021-07-26T06:33:02.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:02.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:02.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:33:02.693 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:33:02.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:02.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:33:02.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:03.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:03.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:03.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:33:03.963 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:33:03.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:03.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:33:04.191 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:04.191+0000 7f6dd55790c0 -1 osd.1 41 log_to_monitors {default=true} 2021-07-26T06:33:04.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:05.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:05.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:05.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:33:05.255 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:33:05.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:05.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:33:05.701 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/157556,v1:127.0.0.1:6811/157556] [v2:127.0.0.1:6812/157556,v1:127.0.0.1:6813/157556] exists,up ca4f7f59-f17f-40a6-b2dd-73208a8eb1cf 2021-07-26T06:33:05.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:33:05.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:33:05.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:33:05.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:33:05.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:33:05.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:33:05.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:33:05.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:33:05.704 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:33:05.704 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:33:05.704 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:33:05.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:33:05.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:33:05.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:33:05.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:33:05.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:33:05.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:33:05.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:33:05.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:33:06.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:33:06.062 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:33:06.063 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:33:06.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:33:06.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:33:06.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:33:06.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691845 2021-07-26T06:33:06.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691845 2021-07-26T06:33:06.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-171798691845' 2021-07-26T06:33:06.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:33:06.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:33:06.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528322 2021-07-26T06:33:06.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528322 2021-07-26T06:33:06.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-171798691845 1-193273528322' 2021-07-26T06:33:06.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:33:06.339 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:33:06.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509464 2021-07-26T06:33:06.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509464 2021-07-26T06:33:06.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-171798691845 1-193273528322 2-64424509464' 2021-07-26T06:33:06.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:33:06.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-171798691845 2021-07-26T06:33:06.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:33:06.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:33:06.460 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-171798691845 2021-07-26T06:33:06.460 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:33:06.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691845 2021-07-26T06:33:06.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2021-07-26T06:33:06.461 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 171798691845 2021-07-26T06:33:06.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:33:06.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691844 -lt 171798691845 2021-07-26T06:33:06.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:33:07.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:33:07.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:33:07.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691844 -lt 171798691845 2021-07-26T06:33:07.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:33:09.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:33:09.000 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:33:09.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691845 -lt 171798691845 2021-07-26T06:33:09.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:33:09.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-193273528322 2021-07-26T06:33:09.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:33:09.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:33:09.280 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-193273528322 2021-07-26T06:33:09.280 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:33:09.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528322 2021-07-26T06:33:09.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2021-07-26T06:33:09.282 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 193273528322 2021-07-26T06:33:09.282 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:33:09.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528322 -lt 193273528322 2021-07-26T06:33:09.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:33:09.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509464 2021-07-26T06:33:09.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:33:09.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:33:09.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509464 2021-07-26T06:33:09.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:33:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509464 2021-07-26T06:33:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2021-07-26T06:33:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509464 2021-07-26T06:33:09.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:33:09.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509464 -lt 64424509464 2021-07-26T06:33:09.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:33:09.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:33:09.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:33:10.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:33:10.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:33:10.218 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:33:10.218 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:33:10.218 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:33:10.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:33:10.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:33:10.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:33:10.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:33:10.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:33:10.479 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:33:10.479 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:33:10.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:33:10.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:33:10.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:33:10.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:591: TEST_auto_repair_bluestore_failed: repair 1.0 2021-07-26T06:33:10.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=1.0 2021-07-26T06:33:10.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 1.0 2021-07-26T06:33:10.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:33:10.811 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:33:10.811 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:33:10.811 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:33:11.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:32:36.570595+0000 2021-07-26T06:33:11.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 1.0 2021-07-26T06:33:11.324 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 1.0 on osd.1 to repair 2021-07-26T06:33:11.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 1.0 2021-07-26T06:32:36.570595+0000 2021-07-26T06:33:11.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.0 2021-07-26T06:33:11.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:32:36.570595+0000 2021-07-26T06:33:11.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:33:11.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:33:11.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:33:11.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:33:11.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:33:11.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:33:11.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:33:11.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:33:11.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:32:36.570595+0000 '>' 2021-07-26T06:32:36.570595+0000 2021-07-26T06:33:11.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:33:12.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:33:12.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:33:12.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:33:12.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:33:12.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:33:12.601 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:33:12.601 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:33:12.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:32:36.570595+0000 '>' 2021-07-26T06:32:36.570595+0000 2021-07-26T06:33:12.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:33:13.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:33:13.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:33:13.877 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:33:13.877 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:33:13.877 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:33:13.877 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:33:13.878 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:33:14.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:32:36.570595+0000 '>' 2021-07-26T06:32:36.570595+0000 2021-07-26T06:33:14.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:33:15.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:33:15.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:33:15.154 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:33:15.154 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:33:15.154 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:33:15.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:33:15.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:33:15.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:32:36.570595+0000 '>' 2021-07-26T06:32:36.570595+0000 2021-07-26T06:33:15.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:33:16.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:33:16.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:33:16.425 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:33:16.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:33:16.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:33:16.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:33:16.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:33:16.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:33:12.211388+0000 '>' 2021-07-26T06:32:36.570595+0000 2021-07-26T06:33:16.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:33:16.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:592: TEST_auto_repair_bluestore_failed: sleep 2 2021-07-26T06:33:18.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: flush_pg_stats 2021-07-26T06:33:18.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:33:18.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:33:18.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:33:18.979 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:33:18.979 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:33:18.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:33:18.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:33:18.980 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:33:19.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691849 2021-07-26T06:33:19.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691849 2021-07-26T06:33:19.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-171798691849' 2021-07-26T06:33:19.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:33:19.126 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:33:19.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528325 2021-07-26T06:33:19.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528325 2021-07-26T06:33:19.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-171798691849 1-193273528325' 2021-07-26T06:33:19.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:33:19.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:33:19.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509467 2021-07-26T06:33:19.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509467 2021-07-26T06:33:19.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-171798691849 1-193273528325 2-64424509467' 2021-07-26T06:33:19.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:33:19.362 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-171798691849 2021-07-26T06:33:19.362 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:33:19.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:33:19.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-171798691849 2021-07-26T06:33:19.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:33:19.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691849 2021-07-26T06:33:19.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 171798691849' 2021-07-26T06:33:19.365 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 171798691849 2021-07-26T06:33:19.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:33:19.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691847 -lt 171798691849 2021-07-26T06:33:19.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:33:20.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:33:20.630 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:33:20.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691849 -lt 171798691849 2021-07-26T06:33:20.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:33:20.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-193273528325 2021-07-26T06:33:20.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:33:20.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:33:20.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-193273528325 2021-07-26T06:33:20.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:33:20.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528325 2021-07-26T06:33:20.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 193273528325' 2021-07-26T06:33:20.910 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 193273528325 2021-07-26T06:33:20.910 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:33:21.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528326 -lt 193273528325 2021-07-26T06:33:21.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:33:21.217 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509467 2021-07-26T06:33:21.217 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:33:21.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:33:21.218 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509467 2021-07-26T06:33:21.218 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:33:21.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509467 2021-07-26T06:33:21.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2021-07-26T06:33:21.220 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509467 2021-07-26T06:33:21.220 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:33:21.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509467 -lt 64424509467 2021-07-26T06:33:21.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2021-07-26T06:33:21.734 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:33:21.734 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 9 0 0 0 0 63 0 0 10 10 active+clean 2021-07-26T06:33:15.784257+0000 21'10 46:132 [1,0] 1 [1,0] 1 21'10 2021-07-26T06:33:12.211388+0000 21'10 2021-07-26T06:33:12.211388+0000 0 2021-07-26T06:33:21.735 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:33:21.735 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:33:21.736 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:33:21.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:596: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2021-07-26T06:33:21.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:596: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2021-07-26T06:33:21.998 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:33:22.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2021-07-26T06:33:22.009 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.214+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=10.216514587s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+wait pruub 11.317797661s@ [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber pg(1.0) scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2021-07-26T06:33:22.010 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.214+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=10.216514587s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+wait pruub 11.317797661s@ [ 1.0: AUTO_REPAIR ] ] PrimaryLog scrubber pg(1.0) _scrub_finish info stats: valid 2021-07-26T06:33:22.010 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.215+0000 7f71a17d7700 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=3.358258009s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair+wait pruub 11.317797661s@ [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber pg(1.0) scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2021-07-26T06:33:22.010 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.569+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=3.121513128s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+wait pruub 11.317797661s@ [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber pg(1.0) scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2021-07-26T06:33:22.011 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.569+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=3.121513128s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+wait pruub 11.317797661s@ [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] PrimaryLog scrubber pg(1.0) _scrub_finish info stats: valid 2021-07-26T06:33:22.011 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:32:36.569+0000 7f71a17d7700 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35 pruub=3.121513128s) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair+wait pruub 11.317797661s@ [ 1.0: REQ_SCRUB ] ] scrubber pg(1.0) scrub_finish 3 error(s) still present after re-scrub 2021-07-26T06:33:22.011 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:33:12.209+0000 7f6db3ebe700 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45 pruub=10.402267456s) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair+wait pruub 11.656287193s@ [ 1.0: REQ_SCRUB ] MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber pg(1.0) scrub_finish before flags: REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2021-07-26T06:33:22.011 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:33:12.209+0000 7f6db3ebe700 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45 pruub=10.402267456s) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair+wait pruub 11.656287193s@ [ 1.0: REQ_SCRUB ] ] PrimaryLog scrubber pg(1.0) _scrub_finish info stats: valid 2021-07-26T06:33:22.012 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:33:12.209+0000 7f6db3ebe700 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45 pruub=3.572732449s) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair+wait pruub 11.656287193s@ [ 1.0: REQ_SCRUB ] ] scrubber pg(1.0) scrub_finish All may be fixed 2021-07-26T06:33:22.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:33:22.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:33:22.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:33:22.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:33:22.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:33:22.015 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:33:22.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:33:22.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:33:22.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:33:22.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:33:22.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:33:22.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:33:22.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:33:22.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:33:22.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:33:22.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:33:22.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:33:22.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:33:22.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:33:22.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:33:22.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:33:22.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:33:22.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:33:22.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:33:22.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:22.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:33:22.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:33:22.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:33:22.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:33:22.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:33:22.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:33:22.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:33:22.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:33:22.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:33:22.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:33:22.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:33:22.138 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:33:22.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:33:22.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:33:22.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:33:22.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:33:22.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:33:22.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:33:22.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:33:22.142 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:33:22.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:33:22.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:33:22.143 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:33:22.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:33:22.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:33:22.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:33:22.143 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:33:22.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:33:22.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:33:22.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:33:22.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:33:22.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:22.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:33:22.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:33:22.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:33:22.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:33:22.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:33:22.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:33:22.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:22.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:33:22.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:33:22.151 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:33:22.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:33:22.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:33:22.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:33:22.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2021-07-26T06:33:22.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:601: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2021-07-26T06:33:22.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:602: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2021-07-26T06:33:22.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2021-07-26T06:33:22.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:33:22.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:33:22.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:33:22.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:33:22.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:33:22.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T06:33:22.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:33:22.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:33:22.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:33:22.582 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:33:22.582 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:22.582 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:33:22.583 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:33:22.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:33:22.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:33:22.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:33:22.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:33:22.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:33:22.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:33:22.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:33:22.886 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:33:22.886 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:33:22.886 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:33:22.886 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:33:22.887 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:22.887 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:33:22.887 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:33:22.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:33:22.888 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:33:22.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:33:22.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:33:22.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:33:22.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:33:22.985 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:33:22.985 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:33:22.985 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:33:22.986 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:33:22.986 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:33:22.986 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:22.986 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:33:22.986 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:33:22.987 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:33:22.987 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:33:23.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:606: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2021-07-26T06:33:23.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:33:23.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:33:23.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:33:23.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:33:23.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:33:23.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:33:23.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:33:23.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:33:23.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:33:23.919 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:33:23.919 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:23.920 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:33:23.920 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:33:23.920 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:33:23.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:33:23.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:33:23.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:610: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2021-07-26T06:33:23.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:610: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2021-07-26T06:33:23.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:611: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:33:23.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:33:23.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:33:23.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:33:23.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:33:23.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:33:23.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:33:23.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:33:23.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:33:23.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:33:23.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:33:23.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:33:23.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:33:23.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:33:23.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:33:23.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:33:23.959 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:33:23.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:33:23.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:33:23.960 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:33:23.960 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:23.961 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:33:23.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:33:23.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:33:23.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:33:23.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:33:23.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:33:23.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:33:23.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:33:23.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:33:23.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:33:23.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:33:23.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:33:23.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:33:23.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:33:23.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:33:23.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=378574cf-884a-4d29-874d-93c9182a79fd 2021-07-26T06:33:23.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 378574cf-884a-4d29-874d-93c9182a79fd' 2021-07-26T06:33:23.965 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 378574cf-884a-4d29-874d-93c9182a79fd 2021-07-26T06:33:23.965 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:33:23.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAzV/5gjwxuORAA/vyLXzm8Fc7cmccbi7PrKg== 2021-07-26T06:33:23.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAzV/5gjwxuORAA/vyLXzm8Fc7cmccbi7PrKg=="}' 2021-07-26T06:33:23.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 378574cf-884a-4d29-874d-93c9182a79fd -i td/osd-scrub-repair/0/new.json 2021-07-26T06:33:24.216 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:33:24.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:33:24.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAzV/5gjwxuORAA/vyLXzm8Fc7cmccbi7PrKg== --osd-uuid 378574cf-884a-4d29-874d-93c9182a79fd 2021-07-26T06:33:24.254 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:24.255+0000 7fd560d2e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:24.255 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:24.257+0000 7fd560d2e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:24.256 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:24.257+0000 7fd560d2e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:24.257 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:24.258+0000 7fd560d2e0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:33:24.257 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:24.258+0000 7fd560d2e0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:33:24.257 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:24.258+0000 7fd560d2e0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:33:24.324 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:24.325+0000 7fd560d2e0c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T06:33:30.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:33:30.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:33:30.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:33:30.352 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:33:30.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:33:30.740 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:33:30.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:33:30.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:33:30.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:33:30.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:33:30.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:33:30.764 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:30.766+0000 7f4f866dd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:30.797 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:30.799+0000 7f4f866dd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:30.798 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:30.800+0000 7f4f866dd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:31.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:33:31.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:33:31.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:33:31.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:33:31.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:33:31.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:31.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:33:31.013 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:33:31.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:31.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:33:31.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:31.328 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:31.330+0000 7f4f866dd0c0 -1 Falling back to public interface 2021-07-26T06:33:32.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:32.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:32.308 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:33:32.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:33:32.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:32.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:33:32.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:33.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:33.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:33.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:33:33.585 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:33:33.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:33.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:33:33.706 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:33.707+0000 7f4f866dd0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:33:33.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:34.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:34.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:34.865 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:33:34.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:33:34.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:34.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:33:35.136 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/160102,v1:127.0.0.1:6803/160102] [v2:127.0.0.1:6804/160102,v1:127.0.0.1:6805/160102] exists,up 378574cf-884a-4d29-874d-93c9182a79fd 2021-07-26T06:33:35.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:33:35.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:33:35.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:33:35.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:610: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2021-07-26T06:33:35.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:611: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:33:35.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:33:35.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:33:35.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:33:35.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:33:35.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:33:35.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:33:35.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:33:35.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:33:35.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:33:35.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:33:35.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:33:35.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:33:35.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:33:35.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:33:35.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:33:35.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:33:35.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:33:35.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:33:35.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:33:35.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:35.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:33:35.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:33:35.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:33:35.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:33:35.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:33:35.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:33:35.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:33:35.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:33:35.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:33:35.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:33:35.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:33:35.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:33:35.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:33:35.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:33:35.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:33:35.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a1093192-36bc-42aa-aa24-dab1dae15ab2 2021-07-26T06:33:35.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a1093192-36bc-42aa-aa24-dab1dae15ab2' 2021-07-26T06:33:35.153 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 a1093192-36bc-42aa-aa24-dab1dae15ab2 2021-07-26T06:33:35.154 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:33:35.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/V/5gr1JTCRAA+q9upmTgOdpKN8mNm3pjJg== 2021-07-26T06:33:35.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/V/5gr1JTCRAA+q9upmTgOdpKN8mNm3pjJg=="}' 2021-07-26T06:33:35.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a1093192-36bc-42aa-aa24-dab1dae15ab2 -i td/osd-scrub-repair/1/new.json 2021-07-26T06:33:35.508 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:33:35.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:33:35.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA/V/5gr1JTCRAA+q9upmTgOdpKN8mNm3pjJg== --osd-uuid a1093192-36bc-42aa-aa24-dab1dae15ab2 2021-07-26T06:33:35.546 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:35.548+0000 7f69469920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:35.548 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:35.549+0000 7f69469920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:35.549 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:35.550+0000 7f69469920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:35.549 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:35.550+0000 7f69469920c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:33:35.550 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:35.550+0000 7f69469920c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:33:35.550 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:35.550+0000 7f69469920c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:33:35.616 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:35.617+0000 7f69469920c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T06:33:41.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:33:41.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:33:41.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:33:41.700 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:33:41.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:33:42.042 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:33:42.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:33:42.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:33:42.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:33:42.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:33:42.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:33:42.066 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:42.067+0000 7f88308210c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:42.068 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:42.069+0000 7f88308210c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:42.069 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:42.070+0000 7f88308210c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:42.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:33:42.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:33:42.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:33:42.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:33:42.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:33:42.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:42.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:33:42.316 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:33:42.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:42.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:33:42.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:42.595 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:42.596+0000 7f88308210c0 -1 Falling back to public interface 2021-07-26T06:33:43.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:43.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:43.582 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:33:43.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:33:43.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:43.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:33:43.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:44.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:44.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:44.859 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:33:44.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:33:44.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:44.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:33:45.092 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:45.094+0000 7f88308210c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:33:45.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:46.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:46.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:46.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:33:46.192 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:33:46.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:46.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:33:46.495 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/161501,v1:127.0.0.1:6811/161501] [v2:127.0.0.1:6812/161501,v1:127.0.0.1:6813/161501] exists,up a1093192-36bc-42aa-aa24-dab1dae15ab2 2021-07-26T06:33:46.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:33:46.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:33:46.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:33:46.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:610: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2021-07-26T06:33:46.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:611: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:33:46.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:33:46.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:33:46.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:33:46.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:33:46.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:33:46.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:33:46.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:33:46.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:33:46.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:33:46.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:33:46.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:33:46.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:33:46.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:33:46.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:33:46.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:33:46.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:33:46.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:33:46.507 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:33:46.507 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:33:46.507 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:46.507 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:33:46.508 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:33:46.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:33:46.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:33:46.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:33:46.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:33:46.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:33:46.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:33:46.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:33:46.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:33:46.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:33:46.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:33:46.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:33:46.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:33:46.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:33:46.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=96fdfd62-1311-43f1-a2ca-10f6de3421e9 2021-07-26T06:33:46.511 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 96fdfd62-1311-43f1-a2ca-10f6de3421e9 2021-07-26T06:33:46.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 96fdfd62-1311-43f1-a2ca-10f6de3421e9' 2021-07-26T06:33:46.512 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:33:46.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKV/5gfRutHhAAi/Zt1PWf+R740k+Mx7nniw== 2021-07-26T06:33:46.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKV/5gfRutHhAAi/Zt1PWf+R740k+Mx7nniw=="}' 2021-07-26T06:33:46.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 96fdfd62-1311-43f1-a2ca-10f6de3421e9 -i td/osd-scrub-repair/2/new.json 2021-07-26T06:33:46.926 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:33:46.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2021-07-26T06:33:46.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBKV/5gfRutHhAAi/Zt1PWf+R740k+Mx7nniw== --osd-uuid 96fdfd62-1311-43f1-a2ca-10f6de3421e9 2021-07-26T06:33:46.965 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:46.966+0000 7fb492ffb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:46.980 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:46.982+0000 7fb492ffb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:46.981 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:46.982+0000 7fb492ffb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:46.982 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:46.983+0000 7fb492ffb0c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:33:46.982 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:46.983+0000 7fb492ffb0c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:33:46.982 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:46.983+0000 7fb492ffb0c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:33:47.056 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:47.057+0000 7fb492ffb0c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2021-07-26T06:33:53.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T06:33:53.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:33:53.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:33:53.086 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:33:53.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:33:53.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:33:53.433 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:33:53.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:33:53.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:33:53.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:33:53.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:33:53.456 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:53.457+0000 7f766da4e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:53.458 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:53.459+0000 7f766da4e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:53.459 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:53.460+0000 7f766da4e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:53.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:33:53.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:33:53.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:33:53.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:33:53.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:33:53.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:53.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:33:53.705 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:33:53.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:53.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:33:53.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:53.983 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:53.984+0000 7f766da4e0c0 -1 Falling back to public interface 2021-07-26T06:33:54.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:54.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:54.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:33:54.973 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:33:54.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:54.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:33:55.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:56.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:56.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:56.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:33:56.308 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:33:56.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:56.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:33:56.417 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:33:56.418+0000 7f766da4e0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:33:56.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:57.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:57.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:57.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:33:57.618 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:33:57.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:57.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:33:57.936 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/162900,v1:127.0.0.1:6819/162900] [v2:127.0.0.1:6820/162900,v1:127.0.0.1:6821/162900] exists,up 96fdfd62-1311-43f1-a2ca-10f6de3421e9 2021-07-26T06:33:57.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:33:57.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:33:57.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:33:57.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2021-07-26T06:33:57.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2021-07-26T06:33:58.439 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'testpool' created 2021-07-26T06:33:58.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:33:59.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:615: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2021-07-26T06:34:00.012 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 1 size to 2 2021-07-26T06:34:00.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:616: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2021-07-26T06:34:00.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:34:00.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:34:00.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:34:00.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:34:00.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:34:00.024 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:34:00.024 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:34:00.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:34:00.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:34:00.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:34:00.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:34:00.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:34:00.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:34:00.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:34:00.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:34:00.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:34:00.445 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:34:00.445 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:34:00.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:34:00.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:00.446 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:34:00.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836487 2021-07-26T06:34:00.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836487 2021-07-26T06:34:00.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487' 2021-07-26T06:34:00.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:00.565 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:34:00.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672965 2021-07-26T06:34:00.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672965 2021-07-26T06:34:00.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2021-07-26T06:34:00.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:00.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:34:00.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509442 2021-07-26T06:34:00.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509442 2021-07-26T06:34:00.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509442' 2021-07-26T06:34:00.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:00.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836487 2021-07-26T06:34:00.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:00.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:34:00.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836487 2021-07-26T06:34:00.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:00.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836487 2021-07-26T06:34:00.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-07-26T06:34:00.863 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836487 2021-07-26T06:34:00.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:34:01.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836487 2021-07-26T06:34:01.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:34:02.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:34:02.153 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:34:02.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836487 2021-07-26T06:34:02.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:34:03.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:34:03.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:34:03.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836487 2021-07-26T06:34:03.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:03.712 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672965 2021-07-26T06:34:03.712 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:03.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:34:03.713 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672965 2021-07-26T06:34:03.713 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:03.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672965 2021-07-26T06:34:03.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-07-26T06:34:03.715 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672965 2021-07-26T06:34:03.715 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:34:03.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672966 -lt 42949672965 2021-07-26T06:34:03.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:03.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509442 2021-07-26T06:34:03.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:03.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:34:03.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509442 2021-07-26T06:34:03.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:03.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509442 2021-07-26T06:34:03.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2021-07-26T06:34:03.999 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509442 2021-07-26T06:34:03.999 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:34:04.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509442 -lt 64424509442 2021-07-26T06:34:04.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:34:04.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:34:04.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:34:04.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:34:04.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:34:04.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:34:04.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:34:04.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:34:04.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:34:04.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:34:04.672 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:34:04.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:34:04.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:34:04.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:34:04.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:34:05.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:34:05.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:34:05.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:34:05.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2021-07-26T06:34:05.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2021-07-26T06:34:05.274 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:621: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2021-07-26T06:34:05.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:621: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2021-07-26T06:34:05.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:34:05.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:621: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2021-07-26T06:34:05.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:34:05.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:621: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2021-07-26T06:34:05.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:34:05.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:621: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2021-07-26T06:34:05.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:34:05.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:621: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2021-07-26T06:34:05.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:34:05.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:621: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2021-07-26T06:34:05.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:34:06.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:621: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2021-07-26T06:34:06.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:34:06.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:621: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2021-07-26T06:34:06.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:34:06.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:621: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2021-07-26T06:34:06.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:34:06.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:621: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2021-07-26T06:34:06.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:34:06.535 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:629: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2021-07-26T06:34:06.535 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T06:34:06.535 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:34:06.535 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool SOMETHING 2021-07-26T06:34:06.536 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:34:06.536 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:34:06.536 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:34:06.536 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:34:06.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:34:06.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:34:06.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:34:07.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:629: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2021-07-26T06:34:07.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:34:07.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:34:07.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:34:07.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:34:07.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2021-07-26T06:34:07.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:34:07.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:34:07.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:34:07.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:34:07.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:34:07.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:34:07.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:34:07.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:34:07.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:34:07.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:34:07.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:34:07.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2021-07-26T06:34:07.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:34:07.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:34:07.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:34:07.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:34:07.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:34:07.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2021-07-26T06:34:09.244 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:34:09.245 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:ff7b1f36:::obj1:head# 2021-07-26T06:34:09.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:34:09.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:34:09.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:34:09.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:34:09.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:34:09.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:34:09.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:34:09.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:34:09.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:34:09.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:34:09.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:34:09.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:34:09.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:34:09.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:34:09.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:34:09.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:34:09.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:34:09.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:34:09.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:34:09.773 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:34:09.773 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:34:09.773 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:34:09.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:34:09.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:34:09.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:34:09.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:34:09.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:34:09.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:34:09.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:34:09.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:34:09.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:34:09.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:34:09.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:34:09.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:34:09.776 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:34:09.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:34:09.777 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:34:09.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:34:09.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:34:09.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:34:09.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:34:09.795 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:09.795+0000 7fef4580d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:09.796 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:09.797+0000 7fef4580d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:09.797 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:09.798+0000 7fef4580d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:10.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:34:10.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:34:10.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:34:10.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:34:10.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:34:10.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:10.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:34:10.052 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:34:10.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:10.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:34:10.313 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:10.314+0000 7fef4580d0c0 -1 Falling back to public interface 2021-07-26T06:34:10.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:11.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:11.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:11.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:34:11.332 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:34:11.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:11.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:34:11.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:12.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:12.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:12.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:34:12.606 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:34:12.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:12.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:34:12.784 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:12.784+0000 7fef4580d0c0 -1 osd.0 20 log_to_monitors {default=true} 2021-07-26T06:34:12.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:13.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:13.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:13.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:34:13.905 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:34:13.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:13.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:34:14.276 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/164572,v1:127.0.0.1:6803/164572] [v2:127.0.0.1:6804/164572,v1:127.0.0.1:6805/164572] exists,up 378574cf-884a-4d29-874d-93c9182a79fd 2021-07-26T06:34:14.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:34:14.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:34:14.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:34:14.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:34:14.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:34:14.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:34:14.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:34:14.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:34:14.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:34:14.278 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:34:14.278 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:34:14.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:34:14.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:34:14.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:34:14.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:34:14.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:34:14.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:34:14.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:34:14.371 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:34:14.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:34:14.644 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:34:14.644 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:34:14.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:34:14.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:14.645 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:34:14.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=103079215106 2021-07-26T06:34:14.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 103079215106 2021-07-26T06:34:14.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-103079215106' 2021-07-26T06:34:14.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:14.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:34:14.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672969 2021-07-26T06:34:14.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672969 2021-07-26T06:34:14.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-103079215106 1-42949672969' 2021-07-26T06:34:14.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:14.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:34:15.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509446 2021-07-26T06:34:15.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509446 2021-07-26T06:34:15.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-103079215106 1-42949672969 2-64424509446' 2021-07-26T06:34:15.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:15.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-103079215106 2021-07-26T06:34:15.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:15.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:34:15.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-103079215106 2021-07-26T06:34:15.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:15.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=103079215106 2021-07-26T06:34:15.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2021-07-26T06:34:15.107 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 103079215106 2021-07-26T06:34:15.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:34:15.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 103079215106 2021-07-26T06:34:15.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:34:16.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:34:16.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:34:16.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 103079215106 2021-07-26T06:34:16.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:34:17.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:34:17.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:34:17.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 103079215106 -lt 103079215106 2021-07-26T06:34:17.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:17.930 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672969 2021-07-26T06:34:17.930 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:17.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:34:17.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672969 2021-07-26T06:34:17.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:17.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672969 2021-07-26T06:34:17.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-07-26T06:34:17.933 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672969 2021-07-26T06:34:17.933 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:34:18.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672969 -lt 42949672969 2021-07-26T06:34:18.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:18.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509446 2021-07-26T06:34:18.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:18.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:34:18.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509446 2021-07-26T06:34:18.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:18.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509446 2021-07-26T06:34:18.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-07-26T06:34:18.206 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509446 2021-07-26T06:34:18.206 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:34:18.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509446 -lt 64424509446 2021-07-26T06:34:18.514 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:34:18.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:34:18.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:34:18.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:34:18.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:34:18.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:34:18.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:34:18.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:34:18.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:34:18.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:34:18.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:34:19.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:34:19.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:34:19.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:34:19.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:34:19.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:34:19.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:34:19.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:34:19.428 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:630: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2021-07-26T06:34:19.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:34:19.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:34:19.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:34:19.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:34:19.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:630: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2021-07-26T06:34:19.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:34:19.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:34:19.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:34:19.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:34:19.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2021-07-26T06:34:19.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:34:19.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:34:19.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:34:19.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:34:19.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:34:19.777 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:34:19.777 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:34:19.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:34:19.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:34:19.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:34:19.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:34:19.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2021-07-26T06:34:19.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:34:19.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:34:19.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:34:19.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:34:19.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:34:19.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2021-07-26T06:34:21.706 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:34:22.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:34:22.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:34:22.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:34:22.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:34:22.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:34:22.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:34:22.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:34:22.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:34:22.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:34:22.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:34:22.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:34:22.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:34:22.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:34:22.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:34:22.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:34:22.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:34:22.232 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:34:22.232 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:34:22.233 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:34:22.233 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:34:22.233 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:34:22.233 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:34:22.233 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:34:22.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:34:22.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:34:22.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:34:22.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:34:22.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:34:22.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:34:22.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:34:22.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:34:22.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:34:22.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:34:22.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:34:22.236 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:34:22.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:34:22.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:34:22.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:34:22.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:34:22.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:34:22.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:34:22.254 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:22.256+0000 7fb6a9d880c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:22.256 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:22.258+0000 7fb6a9d880c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:22.257 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:22.259+0000 7fb6a9d880c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:22.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:34:22.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:34:22.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:34:22.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:34:22.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:34:22.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:22.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:34:22.508 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:34:22.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:22.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:34:22.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:22.780 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:22.781+0000 7fb6a9d880c0 -1 Falling back to public interface 2021-07-26T06:34:23.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:23.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:23.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:34:23.779 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:34:23.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:23.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:34:24.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:25.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:25.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:25.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:34:25.056 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:34:25.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:25.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:34:25.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:25.665 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:25.666+0000 7fb6a9d880c0 -1 osd.1 25 log_to_monitors {default=true} 2021-07-26T06:34:26.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:26.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:26.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:34:26.398 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:34:26.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:26.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:34:26.708 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [10,25) [v2:127.0.0.1:6810/165977,v1:127.0.0.1:6811/165977] [v2:127.0.0.1:6812/165977,v1:127.0.0.1:6813/165977] exists,up a1093192-36bc-42aa-aa24-dab1dae15ab2 2021-07-26T06:34:26.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:34:26.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:34:26.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:34:26.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:34:26.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:34:26.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:34:26.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:34:26.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:34:26.710 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:34:26.710 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:34:26.711 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:34:26.711 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:34:26.711 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:34:26.711 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:34:26.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:34:26.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:34:26.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:34:26.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:34:26.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:34:27.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:34:27.108 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:34:27.108 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:34:27.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:34:27.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:27.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:34:27.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=103079215109 2021-07-26T06:34:27.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 103079215109 2021-07-26T06:34:27.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-103079215109' 2021-07-26T06:34:27.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:27.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:34:27.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=124554051586 2021-07-26T06:34:27.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 124554051586 2021-07-26T06:34:27.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-103079215109 1-124554051586' 2021-07-26T06:34:27.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:27.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:34:27.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509449 2021-07-26T06:34:27.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509449 2021-07-26T06:34:27.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-103079215109 1-124554051586 2-64424509449' 2021-07-26T06:34:27.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:27.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-103079215109 2021-07-26T06:34:27.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:27.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:34:27.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-103079215109 2021-07-26T06:34:27.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:27.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=103079215109 2021-07-26T06:34:27.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2021-07-26T06:34:27.466 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 103079215109 2021-07-26T06:34:27.466 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:34:27.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 103079215108 -lt 103079215109 2021-07-26T06:34:27.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:34:28.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:34:28.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:34:29.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 103079215110 -lt 103079215109 2021-07-26T06:34:29.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:29.007 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-124554051586 2021-07-26T06:34:29.007 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:29.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:34:29.008 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-124554051586 2021-07-26T06:34:29.008 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:29.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=124554051586 2021-07-26T06:34:29.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2021-07-26T06:34:29.010 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 124554051586 2021-07-26T06:34:29.010 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:34:29.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 124554051586 -lt 124554051586 2021-07-26T06:34:29.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:29.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509449 2021-07-26T06:34:29.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:29.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:34:29.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509449 2021-07-26T06:34:29.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:29.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509449 2021-07-26T06:34:29.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2021-07-26T06:34:29.294 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509449 2021-07-26T06:34:29.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:34:29.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509450 -lt 64424509449 2021-07-26T06:34:29.561 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:34:29.561 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:34:29.561 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:34:29.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:34:29.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:34:29.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:34:29.886 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:34:29.886 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:34:29.886 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:34:29.886 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:34:29.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:34:30.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:34:30.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:34:30.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:34:30.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:34:30.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:34:30.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:34:30.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:34:30.471 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2021-07-26T06:34:30.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T06:34:30.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:34:30.472 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool SOMETHING 2021-07-26T06:34:30.472 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:34:30.472 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:34:30.473 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:34:30.473 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:34:30.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:34:30.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:34:30.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:34:31.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2021-07-26T06:34:31.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:34:31.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:34:31.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:34:31.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:34:31.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2021-07-26T06:34:31.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:34:31.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:34:31.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:34:31.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:34:31.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:34:31.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:34:31.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:34:31.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:34:31.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:34:31.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:34:31.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:34:31.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2021-07-26T06:34:31.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:34:31.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:34:31.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:34:31.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:34:31.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:34:31.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2021-07-26T06:34:32.700 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:34:32.701 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:104778fc:::obj2:head# 2021-07-26T06:34:33.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:34:33.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:34:33.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:34:33.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:34:33.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:34:33.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:34:33.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:34:33.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:34:33.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:34:33.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:34:33.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:34:33.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:34:33.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:34:33.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:34:33.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:34:33.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:34:33.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:34:33.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:34:33.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:34:33.226 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:34:33.226 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:34:33.227 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:34:33.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:34:33.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:34:33.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:34:33.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:34:33.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:34:33.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:34:33.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:34:33.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:34:33.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:34:33.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:34:33.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:34:33.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:34:33.230 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:34:33.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:34:33.230 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:34:33.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:34:33.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:34:33.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:34:33.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:34:33.247 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:33.248+0000 7f33c97fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:33.249 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:33.250+0000 7f33c97fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:33.250 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:33.251+0000 7f33c97fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:33.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:34:33.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:34:33.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:34:33.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:34:33.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:34:33.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:33.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:34:33.504 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:34:33.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:33.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:34:33.768 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:33.770+0000 7f33c97fc0c0 -1 Falling back to public interface 2021-07-26T06:34:33.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:34.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:34.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:34.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:34:34.772 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:34:34.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:34.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:34:35.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:36.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:36.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:36.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:34:36.043 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:34:36.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:36.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:34:36.241 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:36.242+0000 7f33c97fc0c0 -1 osd.0 30 log_to_monitors {default=true} 2021-07-26T06:34:36.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:37.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:37.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:37.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:34:37.330 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:34:37.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:37.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:34:37.603 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 34 up_thru 26 down_at 31 last_clean_interval [24,30) [v2:127.0.0.1:6802/167375,v1:127.0.0.1:6803/167375] [v2:127.0.0.1:6804/167375,v1:127.0.0.1:6805/167375] exists,up 378574cf-884a-4d29-874d-93c9182a79fd 2021-07-26T06:34:37.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:34:37.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:34:37.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:34:37.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:34:37.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:34:37.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:34:37.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:34:37.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:34:37.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:34:37.606 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:34:37.606 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:34:37.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:34:37.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:34:37.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:34:37.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:34:37.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:34:37.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:34:37.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:34:37.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:34:37.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:34:37.963 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:34:37.963 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:34:37.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:34:37.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:37.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:34:38.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888066 2021-07-26T06:34:38.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888066 2021-07-26T06:34:38.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888066' 2021-07-26T06:34:38.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:38.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:34:38.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=124554051589 2021-07-26T06:34:38.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 124554051589 2021-07-26T06:34:38.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888066 1-124554051589' 2021-07-26T06:34:38.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:38.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:34:38.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509453 2021-07-26T06:34:38.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509453 2021-07-26T06:34:38.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888066 1-124554051589 2-64424509453' 2021-07-26T06:34:38.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:38.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-146028888066 2021-07-26T06:34:38.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:38.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:34:38.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-146028888066 2021-07-26T06:34:38.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:38.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888066 2021-07-26T06:34:38.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2021-07-26T06:34:38.321 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 146028888066 2021-07-26T06:34:38.321 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:34:38.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 146028888066 2021-07-26T06:34:38.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:34:39.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:34:39.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:34:39.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888066 -lt 146028888066 2021-07-26T06:34:39.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:39.865 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-124554051589 2021-07-26T06:34:39.865 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:39.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:34:39.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-124554051589 2021-07-26T06:34:39.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:39.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=124554051589 2021-07-26T06:34:39.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 124554051589' 2021-07-26T06:34:39.868 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 124554051589 2021-07-26T06:34:39.868 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:34:40.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 124554051589 -lt 124554051589 2021-07-26T06:34:40.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:40.175 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509453 2021-07-26T06:34:40.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:40.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:34:40.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509453 2021-07-26T06:34:40.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:40.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509453 2021-07-26T06:34:40.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2021-07-26T06:34:40.178 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509453 2021-07-26T06:34:40.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:34:40.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509453 -lt 64424509453 2021-07-26T06:34:40.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:34:40.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:34:40.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:34:40.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:34:40.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:34:40.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:34:40.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:34:40.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:34:40.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:34:40.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:34:40.811 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:34:41.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:34:41.112 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:34:41.113 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:34:41.113 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:34:41.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:34:41.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:34:41.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:34:41.438 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2021-07-26T06:34:41.438 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:34:41.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:34:41.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:34:41.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:34:41.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2021-07-26T06:34:41.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:34:41.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:34:41.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:34:41.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:34:41.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2021-07-26T06:34:41.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:34:41.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:34:41.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:34:41.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:34:41.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:34:41.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:34:41.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:34:41.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:34:41.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:34:41.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:34:41.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:34:41.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2021-07-26T06:34:41.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:34:41.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:34:41.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:34:41.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:34:41.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:34:41.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2021-07-26T06:34:43.353 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:34:43.354 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2021-07-26T06:34:43.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:34:43.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:34:43.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:34:43.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:34:43.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:34:43.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:34:43.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:34:43.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:34:43.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:34:43.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:34:43.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:34:43.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:34:43.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:34:43.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:34:43.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:34:43.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:34:43.888 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:34:43.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:34:43.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:34:43.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:34:43.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:34:43.889 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:34:43.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:34:43.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:34:43.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:34:43.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:34:43.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:34:43.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:34:43.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:34:43.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:34:43.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:34:43.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:34:43.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:34:43.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:34:43.892 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:34:43.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:34:43.893 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:34:43.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:34:43.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:34:43.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:34:43.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:34:43.911 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:43.912+0000 7fbc573a60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:43.913 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:43.914+0000 7fbc573a60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:43.914 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:43.915+0000 7fbc573a60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:44.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:34:44.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:34:44.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:34:44.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:34:44.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:34:44.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:44.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:34:44.164 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:34:44.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:44.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:34:44.430 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:44.430+0000 7fbc573a60c0 -1 Falling back to public interface 2021-07-26T06:34:44.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:45.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:45.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:45.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:34:45.475 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:34:45.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:45.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:34:45.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:46.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:46.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:46.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:34:46.754 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:34:46.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:46.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:34:47.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:47.130 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:34:47.132+0000 7fbc573a60c0 -1 osd.1 35 log_to_monitors {default=true} 2021-07-26T06:34:48.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:48.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:48.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:34:48.096 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:34:48.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:48.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:34:48.449 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [29,35) [v2:127.0.0.1:6810/168748,v1:127.0.0.1:6811/168748] [v2:127.0.0.1:6812/168748,v1:127.0.0.1:6813/168748] exists,up a1093192-36bc-42aa-aa24-dab1dae15ab2 2021-07-26T06:34:48.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:34:48.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:34:48.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:34:48.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:34:48.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:34:48.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:34:48.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:34:48.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:34:48.451 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:34:48.452 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:34:48.452 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:34:48.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:34:48.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:34:48.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:34:48.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:34:48.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:34:48.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:34:48.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:34:48.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:34:48.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:34:48.834 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:34:48.834 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:34:48.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:34:48.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:48.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:34:48.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888069 2021-07-26T06:34:48.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888069 2021-07-26T06:34:48.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888069' 2021-07-26T06:34:48.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:48.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:34:49.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=167503724546 2021-07-26T06:34:49.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 167503724546 2021-07-26T06:34:49.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888069 1-167503724546' 2021-07-26T06:34:49.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:49.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:34:49.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509456 2021-07-26T06:34:49.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509456 2021-07-26T06:34:49.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888069 1-167503724546 2-64424509456' 2021-07-26T06:34:49.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:49.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-146028888069 2021-07-26T06:34:49.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:49.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:34:49.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-146028888069 2021-07-26T06:34:49.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:49.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888069 2021-07-26T06:34:49.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 146028888069' 2021-07-26T06:34:49.238 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 146028888069 2021-07-26T06:34:49.238 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:34:49.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888068 -lt 146028888069 2021-07-26T06:34:49.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:34:50.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:34:50.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:34:50.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888069 -lt 146028888069 2021-07-26T06:34:50.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:50.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-167503724546 2021-07-26T06:34:50.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:50.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:34:50.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-167503724546 2021-07-26T06:34:50.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:50.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=167503724546 2021-07-26T06:34:50.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2021-07-26T06:34:50.855 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 167503724546 2021-07-26T06:34:50.856 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:34:51.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 167503724546 -lt 167503724546 2021-07-26T06:34:51.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:51.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509456 2021-07-26T06:34:51.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:51.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:34:51.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509456 2021-07-26T06:34:51.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:51.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509456 2021-07-26T06:34:51.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2021-07-26T06:34:51.190 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509456 2021-07-26T06:34:51.190 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:34:51.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509456 -lt 64424509456 2021-07-26T06:34:51.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:34:51.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:34:51.460 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:34:51.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:34:51.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:34:51.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:34:51.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:34:51.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:34:51.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:34:51.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:34:51.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:34:52.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:34:52.065 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:34:52.066 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:34:52.066 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:34:52.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:34:52.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:34:52.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:34:52.389 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2021-07-26T06:34:52.390 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=testpool 2021-07-26T06:34:52.390 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=obj1 2021-07-26T06:34:52.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map testpool obj1 2021-07-26T06:34:52.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:34:52.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2021-07-26T06:34:52.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2021-07-26T06:34:52.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:34:52.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T06:34:52.699 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool obj1 2021-07-26T06:34:52.699 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:34:52.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2021-07-26T06:34:52.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2021-07-26T06:34:52.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:34:52.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:34:52.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:34:52.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:34:53.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2021-07-26T06:33:58.281821+0000 2021-07-26T06:34:53.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:638: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 deep_scrub 2021-07-26T06:34:53.404 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:34:53.404 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": true, 2021-07-26T06:34:53.404 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:33:13.405863+0000" 2021-07-26T06:34:53.405 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:34:53.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 scrub 2021-07-26T06:34:53.525 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:34:53.525 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:34:53.525 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:33:13.526594+0000" 2021-07-26T06:34:53.525 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:34:53.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2021-07-26T06:33:58.281821+0000 2021-07-26T06:34:53.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.0 2021-07-26T06:34:53.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:33:58.281821+0000 2021-07-26T06:34:53.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:34:53.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:34:53.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:34:53.537 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:34:53.537 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:34:53.537 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:34:53.537 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:34:53.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:34:53.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:33:58.281821+0000 '>' 2021-07-26T06:33:58.281821+0000 2021-07-26T06:34:53.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:34:54.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:34:54.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:34:54.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:34:54.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:34:54.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:34:54.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:34:54.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:34:55.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:33:58.281821+0000 '>' 2021-07-26T06:33:58.281821+0000 2021-07-26T06:34:55.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:34:56.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:34:56.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:34:56.073 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:34:56.073 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:34:56.073 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:34:56.073 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:34:56.074 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:34:56.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:33:58.281821+0000 '>' 2021-07-26T06:33:58.281821+0000 2021-07-26T06:34:56.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:34:57.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:34:57.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:34:57.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:34:57.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:34:57.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:34:57.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:34:57.351 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:34:57.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:33:58.281821+0000 '>' 2021-07-26T06:33:58.281821+0000 2021-07-26T06:34:57.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:34:58.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:34:58.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:34:58.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:34:58.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:34:58.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:34:58.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:34:58.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:34:58.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:34:55.297243+0000 '>' 2021-07-26T06:33:58.281821+0000 2021-07-26T06:34:58.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:34:58.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:643: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2021-07-26T06:34:58.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:34:58.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:34:58.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:34:58.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:34:58.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:34:58.907 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:34:58.907 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:34:58.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:34:58.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:34:58.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:34:59.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:34:59.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:34:59.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:34:59.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:34:59.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:34:59.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:34:59.300 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:34:59.301 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:34:59.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:34:59.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:59.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:34:59.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888072 2021-07-26T06:34:59.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888072 2021-07-26T06:34:59.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888072' 2021-07-26T06:34:59.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:59.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:34:59.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=167503724549 2021-07-26T06:34:59.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 167503724549 2021-07-26T06:34:59.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888072 1-167503724549' 2021-07-26T06:34:59.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:34:59.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:34:59.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509459 2021-07-26T06:34:59.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509459 2021-07-26T06:34:59.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888072 1-167503724549 2-64424509459' 2021-07-26T06:34:59.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:34:59.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-146028888072 2021-07-26T06:34:59.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:34:59.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:34:59.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-146028888072 2021-07-26T06:34:59.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:34:59.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888072 2021-07-26T06:34:59.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 146028888072' 2021-07-26T06:34:59.742 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 146028888072 2021-07-26T06:34:59.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:35:00.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888071 -lt 146028888072 2021-07-26T06:35:00.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:35:01.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:35:01.008 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:35:01.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888072 -lt 146028888072 2021-07-26T06:35:01.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:35:01.296 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-167503724549 2021-07-26T06:35:01.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:35:01.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:35:01.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-167503724549 2021-07-26T06:35:01.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:35:01.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=167503724549 2021-07-26T06:35:01.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 167503724549' 2021-07-26T06:35:01.299 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 167503724549 2021-07-26T06:35:01.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:35:01.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 167503724549 -lt 167503724549 2021-07-26T06:35:01.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:35:01.567 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509459 2021-07-26T06:35:01.567 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:35:01.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:35:01.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509459 2021-07-26T06:35:01.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:35:01.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509459 2021-07-26T06:35:01.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2021-07-26T06:35:01.570 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509459 2021-07-26T06:35:01.570 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:35:01.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509459 -lt 64424509459 2021-07-26T06:35:01.876 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:35:01.876 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:35:01.876 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:35:02.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:35:02.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:35:02.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:35:02.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:35:02.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:35:02.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:35:02.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:35:02.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:35:02.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:35:02.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:35:02.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:35:02.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:35:02.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:35:02.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:35:02.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:35:02.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:644: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2021-07-26T06:35:02.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:35:02.833 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:35:03.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:35:03.143 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:35:03.143 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:35:03.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:35:03.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:35:03.144 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:35:03.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888074 2021-07-26T06:35:03.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888074 2021-07-26T06:35:03.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888074' 2021-07-26T06:35:03.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:35:03.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:35:03.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=167503724551 2021-07-26T06:35:03.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 167503724551 2021-07-26T06:35:03.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888074 1-167503724551' 2021-07-26T06:35:03.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:35:03.379 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:35:03.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509461 2021-07-26T06:35:03.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509461 2021-07-26T06:35:03.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888074 1-167503724551 2-64424509461' 2021-07-26T06:35:03.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:35:03.498 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-146028888074 2021-07-26T06:35:03.498 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:35:03.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:35:03.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-146028888074 2021-07-26T06:35:03.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:35:03.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888074 2021-07-26T06:35:03.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 146028888074' 2021-07-26T06:35:03.501 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 146028888074 2021-07-26T06:35:03.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:35:03.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888073 -lt 146028888074 2021-07-26T06:35:03.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:35:04.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:35:04.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:35:05.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888074 -lt 146028888074 2021-07-26T06:35:05.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:35:05.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-167503724551 2021-07-26T06:35:05.083 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:35:05.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:35:05.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-167503724551 2021-07-26T06:35:05.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:35:05.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=167503724551 2021-07-26T06:35:05.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 167503724551' 2021-07-26T06:35:05.085 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 167503724551 2021-07-26T06:35:05.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:35:05.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 167503724551 -lt 167503724551 2021-07-26T06:35:05.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:35:05.375 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509461 2021-07-26T06:35:05.376 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:35:05.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:35:05.377 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509461 2021-07-26T06:35:05.377 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:35:05.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509461 2021-07-26T06:35:05.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2021-07-26T06:35:05.378 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509461 2021-07-26T06:35:05.379 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:35:05.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509461 -lt 64424509461 2021-07-26T06:35:05.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2021-07-26T06:35:05.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2021-07-26T06:35:05.898 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:35:05.898 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 10 0 0 0 0 56 0 0 10 10 active+clean+inconsistent+failed_repair 2021-07-26T06:34:58.542245+0000 20'10 40:96 [1,0] 1 [1,0] 1 20'10 2021-07-26T06:34:55.297243+0000 20'10 2021-07-26T06:34:55.297243+0000 0 2021-07-26T06:35:05.899 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:35:05.899 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:35:05.900 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:35:05.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2021-07-26T06:35:05.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2021-07-26T06:35:06.162 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:35:06.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:35:06.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:35:06.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:35:06.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:35:06.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:35:06.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:35:06.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:35:06.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:35:06.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:35:06.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:35:06.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:35:06.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:35:06.280 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:35:06.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:35:06.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:35:06.282 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:35:06.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:35:06.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:35:06.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:35:06.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:35:06.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:35:06.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:35:06.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:35:06.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:35:06.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:06.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:35:06.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:35:06.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:35:06.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:35:06.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:35:06.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:35:06.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:35:06.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:35:06.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:35:06.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:35:06.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:35:06.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:35:06.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:35:06.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:35:06.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:35:06.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:35:06.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:35:06.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:35:06.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:35:06.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:35:06.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:35:06.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:35:06.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:35:06.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:35:06.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:35:06.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:35:06.304 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:35:06.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:35:06.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:35:06.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:35:06.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:35:06.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:06.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:35:06.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:35:06.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:35:06.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:35:06.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:35:06.309 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:35:06.309 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:06.309 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:35:06.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:35:06.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:35:06.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:35:06.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:35:06.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:35:06.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2021-07-26T06:35:06.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2021-07-26T06:35:06.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2021-07-26T06:35:06.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2021-07-26T06:35:06.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:35:06.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:35:06.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:35:06.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:35:06.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:35:06.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T06:35:06.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:35:06.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:35:06.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:35:06.717 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:35:06.717 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:06.717 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:35:06.717 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:35:06.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:35:07.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:35:07.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:35:07.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:35:07.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:35:07.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:35:07.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:35:07.004 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:35:07.004 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:35:07.004 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:35:07.004 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:35:07.005 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:07.005 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:35:07.005 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:35:07.005 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:35:07.005 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:35:07.100 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:35:07.100 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:35:07.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:35:07.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:35:07.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:35:07.101 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:35:07.102 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:35:07.102 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:35:07.102 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:35:07.102 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:07.102 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:35:07.103 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:35:07.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:35:07.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:35:07.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2021-07-26T06:35:07.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:35:07.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:35:07.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:35:07.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:35:07.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:35:07.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:35:08.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:35:08.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:35:08.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:35:08.038 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:35:08.039 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:08.039 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:35:08.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:35:08.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:35:08.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:35:08.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2021-07-26T06:35:08.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2021-07-26T06:35:08.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2021-07-26T06:35:08.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2021-07-26T06:35:08.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:35:08.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:35:08.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:35:08.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:35:08.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:35:08.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:35:08.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:35:08.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:35:08.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:35:08.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:35:08.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:35:08.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:35:08.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:35:08.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:35:08.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:35:08.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:35:08.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:35:08.072 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:35:08.072 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:35:08.072 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:08.072 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:35:08.073 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:35:08.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:35:08.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:35:08.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:35:08.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:35:08.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:35:08.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:35:08.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:35:08.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:35:08.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:35:08.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:35:08.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2021-07-26T06:35:08.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:35:08.076 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:35:08.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6548fcd3-eee3-4ced-9698-6986d33109ae 2021-07-26T06:35:08.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6548fcd3-eee3-4ced-9698-6986d33109ae' 2021-07-26T06:35:08.077 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 6548fcd3-eee3-4ced-9698-6986d33109ae 2021-07-26T06:35:08.077 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:35:08.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCcV/5gP6XnBBAAJuAw32SD4hMCW9nsVEs6ow== 2021-07-26T06:35:08.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCcV/5gP6XnBBAAJuAw32SD4hMCW9nsVEs6ow=="}' 2021-07-26T06:35:08.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6548fcd3-eee3-4ced-9698-6986d33109ae -i td/osd-scrub-repair/0/new.json 2021-07-26T06:35:08.334 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:35:08.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:35:08.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCcV/5gP6XnBBAAJuAw32SD4hMCW9nsVEs6ow== --osd-uuid 6548fcd3-eee3-4ced-9698-6986d33109ae 2021-07-26T06:35:08.373 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:08.373+0000 7f5e60b040c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:08.374 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:08.375+0000 7f5e60b040c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:08.375 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:08.376+0000 7f5e60b040c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:08.376 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:08.376+0000 7f5e60b040c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:35:08.376 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:08.376+0000 7f5e60b040c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:35:08.376 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:08.376+0000 7f5e60b040c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:35:08.434 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:08.435+0000 7f5e60b040c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T06:35:14.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:35:14.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:35:14.434 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:35:14.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:35:14.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:35:14.775 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:35:14.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:35:14.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2021-07-26T06:35:14.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:35:14.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:35:14.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:35:14.800 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:14.800+0000 7ff48187a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:14.801 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:14.802+0000 7ff48187a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:14.802 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:14.802+0000 7ff48187a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:15.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:35:15.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:35:15.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:35:15.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:35:15.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:35:15.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:15.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:35:15.068 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:35:15.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:15.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:35:15.318 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:15.319+0000 7ff48187a0c0 -1 Falling back to public interface 2021-07-26T06:35:15.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:16.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:16.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:16.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:35:16.335 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:35:16.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:16.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:35:16.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:17.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:17.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:17.611 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:35:17.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:35:17.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:17.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:35:17.708 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:17.708+0000 7ff48187a0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:35:17.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:18.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:18.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:18.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:35:18.891 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:35:18.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:18.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:35:19.169 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/171716,v1:127.0.0.1:6803/171716] [v2:127.0.0.1:6804/171716,v1:127.0.0.1:6805/171716] exists,up 6548fcd3-eee3-4ced-9698-6986d33109ae 2021-07-26T06:35:19.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:35:19.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:35:19.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:35:19.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2021-07-26T06:35:19.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2021-07-26T06:35:19.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:35:19.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:35:19.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:35:19.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:35:19.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:35:19.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:35:19.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:35:19.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:35:19.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:35:19.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:35:19.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:35:19.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:35:19.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:35:19.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:35:19.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:35:19.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:35:19.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:35:19.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:35:19.180 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:35:19.180 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:19.180 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:35:19.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:35:19.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:35:19.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:35:19.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:35:19.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:35:19.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:35:19.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:35:19.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:35:19.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:35:19.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:35:19.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:35:19.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2021-07-26T06:35:19.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:35:19.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:35:19.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=169b2f9f-f13a-402a-8edb-412fbac34d64 2021-07-26T06:35:19.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 169b2f9f-f13a-402a-8edb-412fbac34d64' 2021-07-26T06:35:19.184 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 169b2f9f-f13a-402a-8edb-412fbac34d64 2021-07-26T06:35:19.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:35:19.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnV/5g+Ko/CxAAgojY5u9ekVGFrnIOGDQBhA== 2021-07-26T06:35:19.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnV/5g+Ko/CxAAgojY5u9ekVGFrnIOGDQBhA=="}' 2021-07-26T06:35:19.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 169b2f9f-f13a-402a-8edb-412fbac34d64 -i td/osd-scrub-repair/1/new.json 2021-07-26T06:35:19.535 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:35:19.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:35:19.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCnV/5g+Ko/CxAAgojY5u9ekVGFrnIOGDQBhA== --osd-uuid 169b2f9f-f13a-402a-8edb-412fbac34d64 2021-07-26T06:35:19.573 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:19.575+0000 7f52600870c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:19.575 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:19.576+0000 7f52600870c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:19.576 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:19.577+0000 7f52600870c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:19.577 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:19.578+0000 7f52600870c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:35:19.577 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:19.578+0000 7f52600870c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:35:19.577 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:19.578+0000 7f52600870c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:35:19.676 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:19.677+0000 7f52600870c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T06:35:25.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:35:25.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:35:25.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:35:25.952 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:35:25.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:35:26.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:35:26.291 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:35:26.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:35:26.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2021-07-26T06:35:26.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:35:26.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:35:26.315 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:26.316+0000 7f72f63f30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:26.316 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:26.318+0000 7f72f63f30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:26.317 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:26.318+0000 7f72f63f30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:26.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:35:26.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:35:26.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:35:26.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:35:26.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:35:26.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:26.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:35:26.561 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:35:26.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:26.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:35:26.831 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:26.833+0000 7f72f63f30c0 -1 Falling back to public interface 2021-07-26T06:35:26.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:27.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:27.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:27.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:35:27.913 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:35:27.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:27.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:35:28.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:29.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:29.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:29.187 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:35:29.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:35:29.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:29.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:35:29.335 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:29.337+0000 7f72f63f30c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:35:29.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:30.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:30.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:30.492 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:35:30.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:35:30.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:30.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:35:30.770 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/173123,v1:127.0.0.1:6811/173123] [v2:127.0.0.1:6812/173123,v1:127.0.0.1:6813/173123] exists,up 169b2f9f-f13a-402a-8edb-412fbac34d64 2021-07-26T06:35:30.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:35:30.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:35:30.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:35:30.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2021-07-26T06:35:30.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2021-07-26T06:35:30.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:35:30.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:35:30.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:35:30.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:35:30.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:35:30.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:35:30.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:35:30.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:35:30.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:35:30.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:35:30.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:35:30.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:35:30.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:35:30.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:35:30.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:35:30.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:35:30.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:35:30.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:35:30.781 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:35:30.781 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:30.781 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:35:30.782 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:35:30.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:35:30.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:35:30.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:35:30.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:35:30.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:35:30.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:35:30.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:35:30.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:35:30.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:35:30.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:35:30.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2021-07-26T06:35:30.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:35:30.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:35:30.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4dd8f6c9-d5a3-4c4f-a0ea-723a47fd1ca7 2021-07-26T06:35:30.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4dd8f6c9-d5a3-4c4f-a0ea-723a47fd1ca7' 2021-07-26T06:35:30.786 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 4dd8f6c9-d5a3-4c4f-a0ea-723a47fd1ca7 2021-07-26T06:35:30.786 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:35:30.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCyV/5gRgogLxAAfmexct0ip3AgqR0tcPnUEQ== 2021-07-26T06:35:30.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCyV/5gRgogLxAAfmexct0ip3AgqR0tcPnUEQ=="}' 2021-07-26T06:35:30.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4dd8f6c9-d5a3-4c4f-a0ea-723a47fd1ca7 -i td/osd-scrub-repair/2/new.json 2021-07-26T06:35:31.163 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:35:31.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2021-07-26T06:35:31.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCyV/5gRgogLxAAfmexct0ip3AgqR0tcPnUEQ== --osd-uuid 4dd8f6c9-d5a3-4c4f-a0ea-723a47fd1ca7 2021-07-26T06:35:31.202 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:31.203+0000 7f62932690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:31.213 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:31.214+0000 7f62932690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:31.214 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:31.215+0000 7f62932690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:31.214 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:31.215+0000 7f62932690c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:35:31.215 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:31.215+0000 7f62932690c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:35:31.215 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:31.215+0000 7f62932690c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:35:31.284 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:31.284+0000 7f62932690c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2021-07-26T06:35:37.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T06:35:37.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:35:37.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:35:37.321 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:35:37.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:35:37.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:35:37.661 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:35:37.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2021-07-26T06:35:37.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:35:37.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:35:37.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:35:37.685 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:37.687+0000 7f90236880c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:37.687 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:37.688+0000 7f90236880c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:37.688 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:37.689+0000 7f90236880c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:37.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:35:37.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:35:37.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:35:37.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:35:37.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:35:37.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:37.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:35:37.942 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:35:37.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:37.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:35:38.206 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:38.208+0000 7f90236880c0 -1 Falling back to public interface 2021-07-26T06:35:38.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:39.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:39.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:39.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:35:39.250 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:35:39.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:39.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:35:39.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:40.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:40.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:40.529 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:35:40.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:35:40.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:40.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:35:40.638 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:40.639+0000 7f90236880c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:35:40.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:41.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:41.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:41.846 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:35:41.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:35:41.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:41.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:35:42.162 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/174522,v1:127.0.0.1:6819/174522] [v2:127.0.0.1:6820/174522,v1:127.0.0.1:6821/174522] exists,up 4dd8f6c9-d5a3-4c4f-a0ea-723a47fd1ca7 2021-07-26T06:35:42.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:35:42.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:35:42.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:35:42.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2021-07-26T06:35:42.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2021-07-26T06:35:42.585 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'testpool' created 2021-07-26T06:35:42.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:35:43.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2021-07-26T06:35:44.233 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 1 size to 2 2021-07-26T06:35:44.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2021-07-26T06:35:44.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:35:44.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:35:44.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:35:44.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:35:44.245 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:35:44.246 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:35:44.246 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:35:44.246 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:35:44.246 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:35:44.247 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:35:44.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:35:44.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:35:44.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:35:44.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:35:44.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:35:44.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:35:44.717 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:35:44.717 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:35:44.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:35:44.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:35:44.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:35:44.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836487 2021-07-26T06:35:44.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836487 2021-07-26T06:35:44.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487' 2021-07-26T06:35:44.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:35:44.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:35:45.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672964 2021-07-26T06:35:45.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672964 2021-07-26T06:35:45.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2021-07-26T06:35:45.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:35:45.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:35:45.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509443 2021-07-26T06:35:45.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509443 2021-07-26T06:35:45.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509443' 2021-07-26T06:35:45.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:35:45.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836487 2021-07-26T06:35:45.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:35:45.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:35:45.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836487 2021-07-26T06:35:45.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:35:45.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836487 2021-07-26T06:35:45.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-07-26T06:35:45.167 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836487 2021-07-26T06:35:45.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:35:45.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836487 2021-07-26T06:35:45.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:35:46.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:35:46.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:35:46.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836487 2021-07-26T06:35:46.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:35:46.831 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672964 2021-07-26T06:35:46.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:35:46.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:35:46.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672964 2021-07-26T06:35:46.833 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:35:46.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672964 2021-07-26T06:35:46.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2021-07-26T06:35:46.834 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672964 2021-07-26T06:35:46.834 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:35:47.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672965 -lt 42949672964 2021-07-26T06:35:47.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:35:47.100 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509443 2021-07-26T06:35:47.100 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:35:47.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:35:47.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509443 2021-07-26T06:35:47.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:35:47.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509443 2021-07-26T06:35:47.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2021-07-26T06:35:47.103 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509443 2021-07-26T06:35:47.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:35:47.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509443 -lt 64424509443 2021-07-26T06:35:47.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:35:47.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:35:47.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:35:47.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:35:47.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:35:47.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:35:47.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:35:47.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:35:47.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:35:47.796 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:35:47.796 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:35:48.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:35:48.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:35:48.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:35:48.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:35:48.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:35:48.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:35:48.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:35:48.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2021-07-26T06:35:48.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2021-07-26T06:35:48.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T06:35:48.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2021-07-26T06:35:48.551 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T06:35:48.551 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:35:48.551 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool SOMETHING 2021-07-26T06:35:48.552 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:35:48.552 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:35:48.552 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:35:48.552 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:35:48.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:35:48.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:35:48.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:35:49.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:35:49.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:35:49.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:35:49.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:35:49.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:35:49.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:35:49.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:35:49.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:35:49.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:35:49.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:35:49.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:35:49.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:35:49.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:35:49.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:35:49.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:35:49.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:35:49.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:35:49.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:35:49.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:35:49.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:35:49.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:35:49.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:35:49.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:35:49.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2021-07-26T06:35:50.886 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:35:50.886 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:eb822e21:::SOMETHING:head# 2021-07-26T06:35:51.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2021-07-26T06:35:51.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:35:51.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:35:51.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:35:51.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:35:51.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:35:51.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:35:51.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:35:51.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:35:51.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:35:51.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:35:51.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:35:51.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:35:51.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:35:51.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:35:51.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:35:51.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:35:51.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:35:51.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:35:51.414 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:35:51.414 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:51.415 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:35:51.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:35:51.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:35:51.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:35:51.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:35:51.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:35:51.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:35:51.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:35:51.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:35:51.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:35:51.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2021-07-26T06:35:51.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:35:51.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:35:51.417 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:35:51.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2021-07-26T06:35:51.418 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:35:51.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:35:51.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:35:51.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:35:51.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:35:51.437 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:51.438+0000 7f610fd190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:51.439 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:51.440+0000 7f610fd190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:51.440 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:51.441+0000 7f610fd190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:51.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:35:51.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:35:51.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:35:51.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:35:51.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:35:51.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:51.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:35:51.691 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:35:51.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:51.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:35:51.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:51.963 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:51.965+0000 7f610fd190c0 -1 Falling back to public interface 2021-07-26T06:35:52.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:52.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:52.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:35:52.955 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:35:52.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:52.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:35:53.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:54.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:54.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:54.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:35:54.232 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:35:54.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:54.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:35:54.449 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:35:54.449+0000 7f610fd190c0 -1 osd.0 21 log_to_monitors {default=true} 2021-07-26T06:35:54.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:55.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:55.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:55.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:35:55.512 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:35:55.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:55.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:35:55.786 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/175989,v1:127.0.0.1:6803/175989] [v2:127.0.0.1:6804/175989,v1:127.0.0.1:6805/175989] exists,up 6548fcd3-eee3-4ced-9698-6986d33109ae 2021-07-26T06:35:55.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:35:55.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:35:55.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:35:55.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:35:55.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:35:55.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:35:55.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:35:55.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:35:55.788 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:35:55.788 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:35:55.789 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:35:55.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:35:55.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:35:55.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:35:55.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:35:55.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:35:55.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:35:55.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:35:55.880 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:35:56.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:35:56.183 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:35:56.183 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:35:56.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:35:56.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:35:56.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:35:56.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182402 2021-07-26T06:35:56.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182402 2021-07-26T06:35:56.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182402' 2021-07-26T06:35:56.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:35:56.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:35:56.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672968 2021-07-26T06:35:56.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672968 2021-07-26T06:35:56.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182402 1-42949672968' 2021-07-26T06:35:56.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:35:56.422 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:35:56.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509446 2021-07-26T06:35:56.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509446 2021-07-26T06:35:56.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182402 1-42949672968 2-64424509446' 2021-07-26T06:35:56.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:35:56.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-107374182402 2021-07-26T06:35:56.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:35:56.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:35:56.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-107374182402 2021-07-26T06:35:56.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:35:56.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182402 2021-07-26T06:35:56.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2021-07-26T06:35:56.542 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 107374182402 2021-07-26T06:35:56.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:35:56.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182402 -lt 107374182402 2021-07-26T06:35:56.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:35:56.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672968 2021-07-26T06:35:56.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:35:56.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:35:56.811 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672968 2021-07-26T06:35:56.811 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:35:56.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672968 2021-07-26T06:35:56.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-07-26T06:35:56.812 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672968 2021-07-26T06:35:56.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:35:57.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672968 -lt 42949672968 2021-07-26T06:35:57.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:35:57.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509446 2021-07-26T06:35:57.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:35:57.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:35:57.081 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509446 2021-07-26T06:35:57.081 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:35:57.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509446 2021-07-26T06:35:57.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-07-26T06:35:57.083 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509446 2021-07-26T06:35:57.083 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:35:57.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509446 -lt 64424509446 2021-07-26T06:35:57.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:35:57.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:35:57.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:35:57.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:35:57.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:35:57.720 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:35:57.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:35:57.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:35:57.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:35:57.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:35:57.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:35:58.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:35:58.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:35:58.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:35:58.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:35:58.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:35:58.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:35:58.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:35:58.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2021-07-26T06:35:58.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=testpool 2021-07-26T06:35:58.347 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:35:58.347 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map testpool SOMETHING 2021-07-26T06:35:58.347 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:35:58.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2021-07-26T06:35:58.654 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2021-07-26T06:35:58.654 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:35:58.654 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:35:58.655 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:35:58.655 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:35:58.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2021-07-26T06:35:58.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2021-07-26T06:35:58.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:35:58.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:35:58.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:35:58.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:35:59.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2021-07-26T06:35:42.476227+0000 2021-07-26T06:35:59.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 scrub 2021-07-26T06:35:59.328 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T06:35:59.328 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T06:35:59.329 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T06:34:19.329913+0000" 2021-07-26T06:35:59.329 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T06:35:59.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2021-07-26T06:35:42.476227+0000 2021-07-26T06:35:59.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.0 2021-07-26T06:35:59.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:35:42.476227+0000 2021-07-26T06:35:59.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:35:59.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:35:59.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:35:59.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:35:59.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:35:59.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:35:59.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:35:59.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:35:59.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:35:42.476227+0000 '>' 2021-07-26T06:35:42.476227+0000 2021-07-26T06:35:59.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:36:00.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:36:00.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:36:00.601 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:36:00.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:36:00.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:36:00.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:36:00.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:36:00.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:35:42.476227+0000 '>' 2021-07-26T06:35:42.476227+0000 2021-07-26T06:36:00.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:36:01.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:36:01.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:36:01.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:36:01.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:36:01.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:36:01.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:36:01.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:36:02.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:35:42.476227+0000 '>' 2021-07-26T06:35:42.476227+0000 2021-07-26T06:36:02.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:36:03.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:36:03.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:36:03.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T06:36:03.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:36:03.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:36:03.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:36:03.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:36:03.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:36:00.088290+0000 '>' 2021-07-26T06:35:42.476227+0000 2021-07-26T06:36:03.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:36:03.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2021-07-26T06:36:03.674 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:36:03.675 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 1 0 0 0 0 7 0 0 1 1 active+clean+inconsistent 2021-07-26T06:36:00.088370+0000 21'1 26:45 [1,0] 1 [1,0] 1 21'1 2021-07-26T06:36:00.088290+0000 0'0 2021-07-26T06:35:42.476227+0000 0 2021-07-26T06:36:03.675 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:36:03.675 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:36:03.676 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:36:03.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 5 2021-07-26T06:36:08.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: wait_for_clean 2021-07-26T06:36:08.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:36:08.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:36:08.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:36:08.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:36:08.689 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:36:08.690 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:36:08.690 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:36:08.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:36:08.691 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:36:08.691 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:36:08.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:36:08.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:36:08.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:36:08.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:36:08.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:36:09.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:36:09.063 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:36:09.063 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:36:09.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:36:09.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:36:09.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:36:09.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182405 2021-07-26T06:36:09.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182405 2021-07-26T06:36:09.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182405' 2021-07-26T06:36:09.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:36:09.181 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:36:09.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672971 2021-07-26T06:36:09.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672971 2021-07-26T06:36:09.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182405 1-42949672971' 2021-07-26T06:36:09.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:36:09.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:36:09.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509450 2021-07-26T06:36:09.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509450 2021-07-26T06:36:09.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182405 1-42949672971 2-64424509450' 2021-07-26T06:36:09.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:36:09.417 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-107374182405 2021-07-26T06:36:09.417 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:36:09.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:36:09.419 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-107374182405 2021-07-26T06:36:09.419 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:36:09.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182405 2021-07-26T06:36:09.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2021-07-26T06:36:09.420 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 107374182405 2021-07-26T06:36:09.421 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:36:09.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182404 -lt 107374182405 2021-07-26T06:36:09.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:36:10.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:36:10.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:36:10.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182406 -lt 107374182405 2021-07-26T06:36:10.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:36:10.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672971 2021-07-26T06:36:10.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:36:10.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:36:10.959 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672971 2021-07-26T06:36:10.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:36:10.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672971 2021-07-26T06:36:10.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2021-07-26T06:36:10.961 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672971 2021-07-26T06:36:10.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:36:11.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672971 -lt 42949672971 2021-07-26T06:36:11.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:36:11.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509450 2021-07-26T06:36:11.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:36:11.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:36:11.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509450 2021-07-26T06:36:11.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:36:11.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509450 2021-07-26T06:36:11.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2021-07-26T06:36:11.271 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509450 2021-07-26T06:36:11.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:36:11.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509450 -lt 64424509450 2021-07-26T06:36:11.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:36:11.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:36:11.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:36:11.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:36:11.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:36:11.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:36:11.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:36:11.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:36:11.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:36:11.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:36:11.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:36:12.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:36:12.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:36:12.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:36:12.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:36:12.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:36:12.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:36:12.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:36:12.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2021-07-26T06:36:12.728 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:36:12.728 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 1 0 0 0 0 7 0 0 1 1 active+clean 2021-07-26T06:36:01.524413+0000 21'1 26:57 [1,0] 1 [1,0] 1 21'1 2021-07-26T06:36:01.110074+0000 21'1 2021-07-26T06:36:01.110074+0000 0 2021-07-26T06:36:12.729 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:36:12.729 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:36:12.730 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:36:12.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2021-07-26T06:36:12.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T06:36:12.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:36:12.741 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool SOMETHING 2021-07-26T06:36:12.741 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:36:12.741 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:36:12.742 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:36:12.742 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:36:13.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:36:13.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:36:13.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:36:13.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2021-07-26T06:36:13.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:36:13.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:36:13.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:36:13.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:36:13.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2021-07-26T06:36:13.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:36:13.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:36:13.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:36:13.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:36:13.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:36:13.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:36:13.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:36:13.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:36:13.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:36:13.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:36:13.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:36:13.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2021-07-26T06:36:13.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:36:13.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:36:13.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:36:13.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:36:13.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:36:13.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2021-07-26T06:36:15.034 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:36:15.034 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:36:15.035 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:36:15.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2021-07-26T06:36:15.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:36:15.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:36:15.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:36:15.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:36:15.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:36:15.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:36:15.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:36:15.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:36:15.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:36:15.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:36:15.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:36:15.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:36:15.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:36:15.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:36:15.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:36:15.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:36:15.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:36:15.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:36:15.317 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:36:15.317 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:36:15.317 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:36:15.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:36:15.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:36:15.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:36:15.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:36:15.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:36:15.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:36:15.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:36:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:36:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:36:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2021-07-26T06:36:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:36:15.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:36:15.321 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:36:15.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2021-07-26T06:36:15.321 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:36:15.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:36:15.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:36:15.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:36:15.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:36:15.335 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:15.336+0000 7f295ee1a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:15.337 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:15.338+0000 7f295ee1a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:15.339 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:15.339+0000 7f295ee1a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:15.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:36:15.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:36:15.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:36:15.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:36:15.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:36:15.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:15.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:36:15.587 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:36:15.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:15.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:36:15.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:15.860 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:15.862+0000 7f295ee1a0c0 -1 Falling back to public interface 2021-07-26T06:36:16.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:16.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:16.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:36:16.854 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:36:16.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:16.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:36:17.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:18.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:18.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:18.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:36:18.132 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:36:18.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:18.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:36:18.400 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:18.401+0000 7f295ee1a0c0 -1 osd.0 26 log_to_monitors {default=true} 2021-07-26T06:36:18.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:19.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:19.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:19.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:36:19.408 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:36:19.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:19.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:36:19.701 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/177993,v1:127.0.0.1:6803/177993] [v2:127.0.0.1:6804/177993,v1:127.0.0.1:6805/177993] exists,up 6548fcd3-eee3-4ced-9698-6986d33109ae 2021-07-26T06:36:19.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:36:19.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:36:19.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:36:19.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:36:19.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:36:19.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:36:19.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:36:19.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:36:19.704 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:36:19.704 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:36:19.704 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:36:19.704 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:36:19.704 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:36:19.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:36:19.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:36:19.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:36:19.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:36:19.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:36:19.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:36:20.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:36:20.106 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:36:20.107 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:36:20.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:36:20.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:36:20.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:36:20.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018882 2021-07-26T06:36:20.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018882 2021-07-26T06:36:20.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018882' 2021-07-26T06:36:20.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:36:20.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:36:20.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672974 2021-07-26T06:36:20.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672974 2021-07-26T06:36:20.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018882 1-42949672974' 2021-07-26T06:36:20.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:36:20.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:36:20.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509453 2021-07-26T06:36:20.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509453 2021-07-26T06:36:20.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018882 1-42949672974 2-64424509453' 2021-07-26T06:36:20.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:36:20.512 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-128849018882 2021-07-26T06:36:20.512 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:36:20.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:36:20.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-128849018882 2021-07-26T06:36:20.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:36:20.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018882 2021-07-26T06:36:20.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2021-07-26T06:36:20.514 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 128849018882 2021-07-26T06:36:20.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:36:20.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 128849018882 2021-07-26T06:36:20.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:36:21.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:36:21.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:36:22.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018882 -lt 128849018882 2021-07-26T06:36:22.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:36:22.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672974 2021-07-26T06:36:22.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:36:22.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:36:22.058 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672974 2021-07-26T06:36:22.058 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:36:22.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672974 2021-07-26T06:36:22.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-07-26T06:36:22.059 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672974 2021-07-26T06:36:22.059 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:36:22.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672974 -lt 42949672974 2021-07-26T06:36:22.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:36:22.325 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509453 2021-07-26T06:36:22.326 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:36:22.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:36:22.328 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509453 2021-07-26T06:36:22.328 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:36:22.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509453 2021-07-26T06:36:22.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2021-07-26T06:36:22.329 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509453 2021-07-26T06:36:22.329 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:36:22.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509453 -lt 64424509453 2021-07-26T06:36:22.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:36:22.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:36:22.603 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:36:22.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:36:22.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:36:22.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:36:22.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:36:22.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:36:22.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:36:22.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:36:22.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:36:23.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:36:23.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:36:23.298 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:36:23.298 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:36:23.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:36:23.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:36:23.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:36:23.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:36:23.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:526: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:36:23.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2021-07-26T06:36:23.649 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:36:00.086+0000 7f72d4d38700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber pg(1.0) scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2021-07-26T06:36:23.650 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:36:00.086+0000 7f72d4d38700 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber pg(1.0) scrub_finish Try to auto repair after scrub errors 2021-07-26T06:36:23.651 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:36:00.086+0000 7f72d4d38700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] PrimaryLog scrubber pg(1.0) _scrub_finish info stats: valid 2021-07-26T06:36:23.651 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:36:01.108+0000 7f72d4d38700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber pg(1.0) scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2021-07-26T06:36:23.652 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:36:01.108+0000 7f72d4d38700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] PrimaryLog scrubber pg(1.0) _scrub_finish info stats: valid 2021-07-26T06:36:23.652 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:36:01.108+0000 7f72d4d38700 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber pg(1.0) scrub_finish All may be fixed 2021-07-26T06:36:23.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2021-07-26T06:36:23.654 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2021-07-26T06:36:23.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: COUNT=1 2021-07-26T06:36:23.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: test 1 = 1 2021-07-26T06:36:23.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:36:23.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:36:23.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:36:23.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:36:23.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:36:23.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:36:23.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:36:23.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:36:23.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:36:23.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:36:23.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:36:23.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:36:23.920 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:36:23.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:36:23.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:36:23.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:36:23.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:36:23.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:36:23.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:36:23.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:36:23.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:36:23.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:36:23.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:36:23.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:36:23.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:36:23.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:36:23.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:36:23.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:36:23.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:36:23.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:36:23.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:36:23.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:36:23.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:36:23.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:36:23.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:36:23.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:36:23.938 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:36:23.938 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:36:23.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:36:23.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:36:23.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:36:23.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:36:23.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:36:23.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:36:23.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:36:23.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:36:23.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:36:23.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:36:23.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:36:23.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:36:23.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:36:23.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:36:23.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:36:23.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:36:23.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:36:23.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:36:23.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:36:23.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:36:23.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:36:23.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:36:23.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:36:23.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:36:23.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:36:23.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:36:23.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:36:23.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:36:23.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:36:23.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:36:23.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:36:23.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:36:23.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2021-07-26T06:36:23.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2021-07-26T06:36:23.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: local poolname=testpool 2021-07-26T06:36:23.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:391: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2021-07-26T06:36:23.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:36:23.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:36:23.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:36:23.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:36:23.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:36:23.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T06:36:24.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:36:24.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:36:24.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:36:24.303 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:36:24.303 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:36:24.304 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:36:24.304 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:36:24.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:36:24.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:36:24.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:36:24.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:36:24.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:36:24.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:36:24.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:36:24.557 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:36:24.557 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:36:24.557 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:36:24.557 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:36:24.558 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:36:24.558 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:36:24.558 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:36:24.558 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:36:24.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:36:24.655 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:36:24.730 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:36:24.730 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:36:24.730 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:36:24.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:36:24.731 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:36:24.731 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:36:24.731 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:36:24.732 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:36:24.732 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:36:24.732 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:36:24.732 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:36:24.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:36:24.733 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:36:24.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:392: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2021-07-26T06:36:24.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:36:24.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:36:24.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:36:24.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:36:24.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:36:24.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:36:25.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:36:25.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:36:25.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:36:25.608 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:36:25.609 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:36:25.609 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:36:25.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:36:25.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:36:25.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:36:25.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:36:25.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2021-07-26T06:36:25.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2021-07-26T06:36:25.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:36:25.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:36:25.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:36:25.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:36:25.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:36:25.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:36:25.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:36:25.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:36:25.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:36:25.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:36:25.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:36:25.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:36:25.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:36:25.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:36:25.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:36:25.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:36:25.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:36:25.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:36:25.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:36:25.642 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:36:25.642 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:36:25.643 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:36:25.643 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:36:25.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:36:25.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:36:25.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:36:25.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:36:25.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:36:25.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:36:25.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:36:25.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:36:25.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:36:25.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:36:25.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:36:25.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:36:25.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:36:25.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c00e533-4d59-4dfe-9871-4461f609206d 2021-07-26T06:36:25.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8c00e533-4d59-4dfe-9871-4461f609206d' 2021-07-26T06:36:25.647 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 8c00e533-4d59-4dfe-9871-4461f609206d 2021-07-26T06:36:25.647 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:36:25.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpV/5ghSXuJhAA6sWRE2iKeAdNWqYtFt33yg== 2021-07-26T06:36:25.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpV/5ghSXuJhAA6sWRE2iKeAdNWqYtFt33yg=="}' 2021-07-26T06:36:25.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c00e533-4d59-4dfe-9871-4461f609206d -i td/osd-scrub-repair/0/new.json 2021-07-26T06:36:26.004 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:36:26.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:36:26.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDpV/5ghSXuJhAA6sWRE2iKeAdNWqYtFt33yg== --osd-uuid 8c00e533-4d59-4dfe-9871-4461f609206d 2021-07-26T06:36:26.043 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:26.045+0000 7fb8770cb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:26.045 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:26.046+0000 7fb8770cb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:26.046 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:26.047+0000 7fb8770cb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:26.047 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:26.048+0000 7fb8770cb0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:36:26.047 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:26.048+0000 7fb8770cb0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:36:26.047 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:26.048+0000 7fb8770cb0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:36:26.096 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:26.097+0000 7fb8770cb0c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T06:36:32.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:36:32.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:36:32.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:36:32.006 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:36:32.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:36:32.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:36:32.348 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:36:32.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:36:32.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:36:32.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:36:32.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:36:32.371 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:32.372+0000 7f6d750d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:32.373 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:32.374+0000 7f6d750d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:32.374 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:32.375+0000 7f6d750d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:32.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:36:32.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:36:32.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:36:32.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:36:32.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:36:32.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:32.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:36:32.657 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:36:32.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:32.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:36:32.891 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:32.893+0000 7f6d750d80c0 -1 Falling back to public interface 2021-07-26T06:36:32.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:33.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:33.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:33.923 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:36:33.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:36:33.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:33.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:36:34.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:35.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:35.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:35.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:36:35.207 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:36:35.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:35.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:36:35.306 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:35.307+0000 7f6d750d80c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:36:35.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:36.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:36.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:36.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:36:36.514 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:36:36.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:36.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:36:36.787 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/180133,v1:127.0.0.1:6803/180133] [v2:127.0.0.1:6804/180133,v1:127.0.0.1:6805/180133] exists,up 8c00e533-4d59-4dfe-9871-4461f609206d 2021-07-26T06:36:36.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:36:36.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:36:36.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:36:36.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2021-07-26T06:36:36.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:36:36.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:36:36.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:36:36.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:36:36.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:36:36.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:36:36.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:36:36.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:36:36.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:36:36.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:36:36.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:36:36.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:36:36.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:36:36.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:36:36.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:36:36.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:36:36.797 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:36:36.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:36:36.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:36:36.798 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:36:36.798 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:36:36.798 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:36:36.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:36:36.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:36:36.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:36:36.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:36:36.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:36:36.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:36:36.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:36:36.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:36:36.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:36:36.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:36:36.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:36:36.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:36:36.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:36:36.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:36:36.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d1fb5fc5-1898-4dd1-bb8a-5159b62368b6 2021-07-26T06:36:36.802 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 d1fb5fc5-1898-4dd1-bb8a-5159b62368b6 2021-07-26T06:36:36.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d1fb5fc5-1898-4dd1-bb8a-5159b62368b6' 2021-07-26T06:36:36.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:36:36.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0V/5g+h8VMBAAGNHrHqCO43VKd7axCFwCdw== 2021-07-26T06:36:36.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0V/5g+h8VMBAAGNHrHqCO43VKd7axCFwCdw=="}' 2021-07-26T06:36:36.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d1fb5fc5-1898-4dd1-bb8a-5159b62368b6 -i td/osd-scrub-repair/1/new.json 2021-07-26T06:36:37.175 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:36:37.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:36:37.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD0V/5g+h8VMBAAGNHrHqCO43VKd7axCFwCdw== --osd-uuid d1fb5fc5-1898-4dd1-bb8a-5159b62368b6 2021-07-26T06:36:37.213 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:37.214+0000 7fe01e1520c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:37.214 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:37.216+0000 7fe01e1520c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:37.215 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:37.217+0000 7fe01e1520c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:37.216 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:37.217+0000 7fe01e1520c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:36:37.217 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:37.217+0000 7fe01e1520c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:36:37.217 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:37.217+0000 7fe01e1520c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:36:37.274 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:37.275+0000 7fe01e1520c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T06:36:43.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:36:43.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:36:43.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:36:43.405 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:36:43.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:36:43.742 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:36:43.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:36:43.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:36:43.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:36:43.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:36:43.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:36:43.766 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:43.767+0000 7f73f6b2f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:43.767 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:43.769+0000 7f73f6b2f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:43.769 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:43.770+0000 7f73f6b2f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:44.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:36:44.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:36:44.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:36:44.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:36:44.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:36:44.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:44.012 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:36:44.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:36:44.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:44.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:36:44.284 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:44.285+0000 7f73f6b2f0c0 -1 Falling back to public interface 2021-07-26T06:36:44.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:45.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:45.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:45.289 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:36:45.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:36:45.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:45.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:36:45.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:46.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:46.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:46.566 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:36:46.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:36:46.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:46.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:36:46.764 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:46.765+0000 7f73f6b2f0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:36:46.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:47.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:47.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:47.908 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:36:47.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:36:47.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:47.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:36:48.182 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/181532,v1:127.0.0.1:6811/181532] [v2:127.0.0.1:6812/181532,v1:127.0.0.1:6813/181532] exists,up d1fb5fc5-1898-4dd1-bb8a-5159b62368b6 2021-07-26T06:36:48.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:36:48.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:36:48.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:36:48.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2021-07-26T06:36:48.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:36:48.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:36:48.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:36:48.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:36:48.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:36:48.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:36:48.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:36:48.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:36:48.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:36:48.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:36:48.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:36:48.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:36:48.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:36:48.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:36:48.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:36:48.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:36:48.200 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:36:48.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:36:48.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:36:48.201 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:36:48.202 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:36:48.202 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:36:48.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:36:48.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:36:48.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:36:48.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:36:48.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:36:48.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:36:48.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:36:48.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:36:48.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:36:48.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:36:48.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:36:48.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:36:48.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:36:48.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:36:48.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fdfbb46d-ae83-4018-adfd-99cb07829cb2 2021-07-26T06:36:48.206 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 fdfbb46d-ae83-4018-adfd-99cb07829cb2 2021-07-26T06:36:48.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fdfbb46d-ae83-4018-adfd-99cb07829cb2' 2021-07-26T06:36:48.207 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:36:48.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAAWP5gE4D7CxAAABSkcZAqXUXuJvAX9otpbg== 2021-07-26T06:36:48.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAAWP5gE4D7CxAAABSkcZAqXUXuJvAX9otpbg=="}' 2021-07-26T06:36:48.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fdfbb46d-ae83-4018-adfd-99cb07829cb2 -i td/osd-scrub-repair/2/new.json 2021-07-26T06:36:48.715 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:36:48.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2021-07-26T06:36:48.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAAWP5gE4D7CxAAABSkcZAqXUXuJvAX9otpbg== --osd-uuid fdfbb46d-ae83-4018-adfd-99cb07829cb2 2021-07-26T06:36:48.753 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:48.754+0000 7f10fbac30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:48.754 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:48.756+0000 7f10fbac30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:48.755 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:48.756+0000 7f10fbac30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:48.755 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:48.757+0000 7f10fbac30c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:36:48.756 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:48.757+0000 7f10fbac30c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:36:48.756 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:48.757+0000 7f10fbac30c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:36:48.817 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:48.818+0000 7f10fbac30c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2021-07-26T06:36:55.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T06:36:55.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:36:55.112 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:36:55.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:36:55.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:36:55.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:36:55.451 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:36:55.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:36:55.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:36:55.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:36:55.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:36:55.474 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:55.475+0000 7f23df0c20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:55.476 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:55.477+0000 7f23df0c20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:55.477 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:55.477+0000 7f23df0c20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:36:55.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:36:55.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:36:55.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:36:55.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:36:55.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:36:55.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:55.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:36:55.722 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:36:55.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:55.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:36:55.992 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:55.993+0000 7f23df0c20c0 -1 Falling back to public interface 2021-07-26T06:36:56.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:57.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:57.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:57.029 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:36:57.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:36:57.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:57.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:36:57.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:58.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:58.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:58.305 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:36:58.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:36:58.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:58.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:36:58.486 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:36:58.486+0000 7f23df0c20c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:36:58.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:59.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:59.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:59.603 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:36:59.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:36:59.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:59.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:36:59.875 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/182931,v1:127.0.0.1:6819/182931] [v2:127.0.0.1:6820/182931,v1:127.0.0.1:6821/182931] exists,up fdfbb46d-ae83-4018-adfd-99cb07829cb2 2021-07-26T06:36:59.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:36:59.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:36:59.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:36:59.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2021-07-26T06:36:59.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2021-07-26T06:37:00.275 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'testpool' created 2021-07-26T06:37:00.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:37:01.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2021-07-26T06:37:01.916 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 1 size to 2 2021-07-26T06:37:01.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2021-07-26T06:37:01.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:37:01.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:37:01.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:37:01.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:37:01.930 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:37:01.930 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:37:01.931 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:37:01.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:37:01.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:37:01.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:37:02.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:37:02.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:37:02.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:37:02.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:37:02.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:37:02.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:37:02.352 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:37:02.352 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:37:02.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:37:02.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:02.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:37:02.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836487 2021-07-26T06:37:02.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836487 2021-07-26T06:37:02.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487' 2021-07-26T06:37:02.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:02.520 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:37:02.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672964 2021-07-26T06:37:02.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672964 2021-07-26T06:37:02.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2021-07-26T06:37:02.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:02.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:37:02.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509443 2021-07-26T06:37:02.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509443 2021-07-26T06:37:02.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509443' 2021-07-26T06:37:02.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:02.815 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836487 2021-07-26T06:37:02.816 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:02.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:37:02.817 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836487 2021-07-26T06:37:02.817 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:02.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836487 2021-07-26T06:37:02.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-07-26T06:37:02.818 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836487 2021-07-26T06:37:02.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:37:03.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836487 2021-07-26T06:37:03.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:37:04.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:37:04.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:37:04.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836487 2021-07-26T06:37:04.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:04.361 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672964 2021-07-26T06:37:04.361 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:04.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:37:04.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672964 2021-07-26T06:37:04.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:04.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672964 2021-07-26T06:37:04.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2021-07-26T06:37:04.364 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672964 2021-07-26T06:37:04.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:37:04.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672965 -lt 42949672964 2021-07-26T06:37:04.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:04.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509443 2021-07-26T06:37:04.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:04.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:37:04.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509443 2021-07-26T06:37:04.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:04.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509443 2021-07-26T06:37:04.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2021-07-26T06:37:04.642 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509443 2021-07-26T06:37:04.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:37:04.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509443 -lt 64424509443 2021-07-26T06:37:04.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:37:04.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:37:04.910 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:37:05.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:37:05.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:37:05.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:37:05.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:37:05.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:37:05.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:37:05.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:37:05.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:37:05.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:37:05.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:37:05.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:37:05.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:37:05.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:37:05.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:37:05.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:37:05.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2021-07-26T06:37:05.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2021-07-26T06:37:05.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T06:37:06.043 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2021-07-26T06:37:06.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T06:37:06.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:37:06.044 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool SOMETHING 2021-07-26T06:37:06.045 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:37:06.045 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:37:06.045 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:37:06.045 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:37:06.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:37:06.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:37:06.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:37:06.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:37:06.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:37:06.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:37:06.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:37:06.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:37:06.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:37:06.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:37:06.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:37:06.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:37:06.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:37:06.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:37:06.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:37:06.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:37:06.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:37:06.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:37:06.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:37:06.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:37:06.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:37:06.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:37:06.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:37:06.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:37:06.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:37:06.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:37:06.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2021-07-26T06:37:08.527 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:37:08.527 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:eb822e21:::SOMETHING:head# 2021-07-26T06:37:09.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:37:09.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:37:09.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:37:09.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:37:09.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:37:09.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:37:09.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:37:09.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:37:09.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:37:09.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:37:09.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:37:09.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:37:09.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:37:09.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:37:09.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:37:09.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:37:09.053 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:37:09.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:37:09.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:37:09.054 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:37:09.054 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:09.054 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:09.055 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:37:09.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:37:09.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:37:09.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:37:09.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:37:09.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:37:09.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:37:09.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:37:09.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:37:09.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:37:09.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:37:09.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:37:09.057 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:37:09.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:37:09.058 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:37:09.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:37:09.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:37:09.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:37:09.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:37:09.074 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:09.075+0000 7f94f24720c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:09.076 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:09.077+0000 7f94f24720c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:09.077 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:09.078+0000 7f94f24720c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:09.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:37:09.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:37:09.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:37:09.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:37:09.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:37:09.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:09.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:37:09.329 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:37:09.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:09.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:09.602 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:09.603+0000 7f94f24720c0 -1 Falling back to public interface 2021-07-26T06:37:09.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:10.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:10.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:10.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:37:10.633 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:37:10.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:10.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:10.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:11.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:11.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:11.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:37:11.907 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:37:11.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:11.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:12.059 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:12.059+0000 7f94f24720c0 -1 osd.0 21 log_to_monitors {default=true} 2021-07-26T06:37:12.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:13.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:13.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:13.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:37:13.185 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:37:13.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:13.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:13.539 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/184399,v1:127.0.0.1:6803/184399] [v2:127.0.0.1:6804/184399,v1:127.0.0.1:6805/184399] exists,up 8c00e533-4d59-4dfe-9871-4461f609206d 2021-07-26T06:37:13.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:37:13.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:37:13.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:37:13.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:37:13.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:37:13.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:37:13.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:37:13.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:37:13.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:37:13.541 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:37:13.541 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:37:13.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:37:13.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:37:13.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:37:13.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:37:13.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:37:13.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:37:13.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:37:13.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:37:13.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:37:13.899 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:37:13.899 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:37:13.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:37:13.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:13.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:37:14.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182402 2021-07-26T06:37:14.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182402 2021-07-26T06:37:14.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182402' 2021-07-26T06:37:14.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:14.018 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:37:14.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672968 2021-07-26T06:37:14.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672968 2021-07-26T06:37:14.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182402 1-42949672968' 2021-07-26T06:37:14.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:14.137 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:37:14.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509446 2021-07-26T06:37:14.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509446 2021-07-26T06:37:14.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182402 1-42949672968 2-64424509446' 2021-07-26T06:37:14.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:14.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-107374182402 2021-07-26T06:37:14.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:14.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:37:14.257 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-107374182402 2021-07-26T06:37:14.258 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:14.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182402 2021-07-26T06:37:14.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2021-07-26T06:37:14.259 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 107374182402 2021-07-26T06:37:14.259 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:37:14.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182402 -lt 107374182402 2021-07-26T06:37:14.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:14.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672968 2021-07-26T06:37:14.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:14.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:37:14.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672968 2021-07-26T06:37:14.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:14.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672968 2021-07-26T06:37:14.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-07-26T06:37:14.541 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672968 2021-07-26T06:37:14.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:37:14.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672968 -lt 42949672968 2021-07-26T06:37:14.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:14.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509446 2021-07-26T06:37:14.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:14.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:37:14.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509446 2021-07-26T06:37:14.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:14.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509446 2021-07-26T06:37:14.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-07-26T06:37:14.827 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509446 2021-07-26T06:37:14.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:37:15.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509445 -lt 64424509446 2021-07-26T06:37:15.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:37:16.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:37:16.136 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:37:16.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509447 -lt 64424509446 2021-07-26T06:37:16.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:37:16.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:37:16.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:37:16.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:37:16.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:37:16.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:37:16.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:37:16.745 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:37:16.745 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:37:16.745 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:37:16.745 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:37:17.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:37:17.007 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:37:17.008 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:37:17.008 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:37:17.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:37:17.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:37:17.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:37:17.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2021-07-26T06:37:17.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=testpool 2021-07-26T06:37:17.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:37:17.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map testpool SOMETHING 2021-07-26T06:37:17.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:37:17.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2021-07-26T06:37:17.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2021-07-26T06:37:17.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:37:17.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:37:17.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:37:17.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:37:17.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2021-07-26T06:37:17.915 INFO:tasks.workunit.client.0.smithi071.stdout:Affected PG 1.0 w/ primary 1 2021-07-26T06:37:17.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2021-07-26T06:37:17.917 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2021-07-26T06:37:17.917 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:37:17.917 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:37:17.917 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:37:17.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:37:18.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2021-07-26T06:37:00.188464+0000 2021-07-26T06:37:18.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2021-07-26T06:37:18.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:325: initiate_and_fetch_state: local the_osd=osd.1 2021-07-26T06:37:18.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: local pgid=1.0 2021-07-26T06:37:18.182 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2021-07-26T06:37:18.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T06:37:18.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:37:18.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:37:18.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T06:37:18.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: local last_scrub=2021-07-26T06:37:00.188464+0000 2021-07-26T06:37:18.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2021-07-26T06:37:18.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local daemon=osd 2021-07-26T06:37:18.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: set_config: local id=1 2021-07-26T06:37:18.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: local config=osd_scrub_sleep 2021-07-26T06:37:18.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: local value=3.0 2021-07-26T06:37:18.453 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: jq 'has("success")' 2021-07-26T06:37:18.453 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: get_asok_path osd.1 2021-07-26T06:37:18.454 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:37:18.454 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:37:18.454 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:37:18.454 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:18.455 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:18.455 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.1.asok 2021-07-26T06:37:18.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.128899/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2021-07-26T06:37:18.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: test true == true 2021-07-26T06:37:18.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2021-07-26T06:37:18.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local daemon=osd 2021-07-26T06:37:18.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: set_config: local id=1 2021-07-26T06:37:18.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: local config=osd_scrub_auto_repair 2021-07-26T06:37:18.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: local value=true 2021-07-26T06:37:18.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: jq 'has("success")' 2021-07-26T06:37:18.557 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: get_asok_path osd.1 2021-07-26T06:37:18.557 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:37:18.558 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:37:18.558 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:37:18.558 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:18.558 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:18.559 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.1.asok 2021-07-26T06:37:18.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.128899/ceph-osd.1.asok config set osd_scrub_auto_repair true 2021-07-26T06:37:18.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: test true == true 2021-07-26T06:37:18.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:332: initiate_and_fetch_state: flush_pg_stats 2021-07-26T06:37:18.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:37:18.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:37:18.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:37:18.963 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:37:18.963 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:37:18.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:37:18.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:18.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:37:19.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182404 2021-07-26T06:37:19.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182404 2021-07-26T06:37:19.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182404' 2021-07-26T06:37:19.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:19.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:37:19.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672970 2021-07-26T06:37:19.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672970 2021-07-26T06:37:19.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182404 1-42949672970' 2021-07-26T06:37:19.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:19.200 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:37:19.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509448 2021-07-26T06:37:19.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509448 2021-07-26T06:37:19.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182404 1-42949672970 2-64424509448' 2021-07-26T06:37:19.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:19.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-107374182404 2021-07-26T06:37:19.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:19.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:37:19.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-107374182404 2021-07-26T06:37:19.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:19.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182404 2021-07-26T06:37:19.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2021-07-26T06:37:19.321 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 107374182404 2021-07-26T06:37:19.321 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:37:19.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182403 -lt 107374182404 2021-07-26T06:37:19.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:37:20.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:37:20.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:37:20.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182404 -lt 107374182404 2021-07-26T06:37:20.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:20.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672970 2021-07-26T06:37:20.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:20.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:37:20.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672970 2021-07-26T06:37:20.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:20.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672970 2021-07-26T06:37:20.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2021-07-26T06:37:20.864 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672970 2021-07-26T06:37:20.864 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:37:21.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672970 -lt 42949672970 2021-07-26T06:37:21.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:21.130 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509448 2021-07-26T06:37:21.130 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:21.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:37:21.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509448 2021-07-26T06:37:21.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:21.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509448 2021-07-26T06:37:21.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2021-07-26T06:37:21.133 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509448 2021-07-26T06:37:21.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:37:21.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509449 -lt 64424509448 2021-07-26T06:37:21.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: date --rfc-3339=ns 2021-07-26T06:37:21.440 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26 06:37:21.441657635+00:00 2021-07-26T06:37:21.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: get_asok_path osd.1 2021-07-26T06:37:21.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:37:21.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:37:21.442 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:37:21.442 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:21.442 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:21.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.1.asok 2021-07-26T06:37:21.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.128899/ceph-osd.1.asok deep_scrub 1.0 2021-07-26T06:37:21.544 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:337: initiate_and_fetch_state: get_asok_path osd.1 2021-07-26T06:37:21.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:37:21.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:37:21.545 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:37:21.545 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:21.546 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:21.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.1.asok 2021-07-26T06:37:21.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:337: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.128899/ceph-osd.1.asok scrub 1.0 2021-07-26T06:37:21.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i=0 )) 2021-07-26T06:37:21.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2021-07-26T06:37:21.652 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2021-07-26T06:37:21.652 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2021-07-26T06:37:21.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2021-07-26T06:37:21.774 INFO:tasks.workunit.client.0.smithi071.stdout:{"deep":true,"stamp":"2021-07-19T06:35:41.536972+0000"}{"deep":false,"stamp":"2021-07-19T06:35:41.644667+0000"}0 ) state now: "active+clean" 2021-07-26T06:37:21.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2021-07-26T06:37:21.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2021-07-26T06:37:21.775 INFO:tasks.workunit.client.0.smithi071.stdout:"active+clean" 2021-07-26T06:37:21.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2021-07-26T06:37:21.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 0 == 4 ']' 2021-07-26T06:37:21.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2021-07-26T06:37:22.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2021-07-26T06:37:22.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2021-07-26T06:37:22.075 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2021-07-26T06:37:22.076 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2021-07-26T06:37:22.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2021-07-26T06:37:22.197 INFO:tasks.workunit.client.0.smithi071.stdout:1 ) state now: "active+clean" 2021-07-26T06:37:22.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2021-07-26T06:37:22.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2021-07-26T06:37:22.198 INFO:tasks.workunit.client.0.smithi071.stdout:"active+clean" 2021-07-26T06:37:22.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2021-07-26T06:37:22.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 1 == 4 ']' 2021-07-26T06:37:22.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2021-07-26T06:37:22.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2021-07-26T06:37:22.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2021-07-26T06:37:22.498 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2021-07-26T06:37:22.498 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2021-07-26T06:37:22.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2021-07-26T06:37:22.621 INFO:tasks.workunit.client.0.smithi071.stdout:2 ) state now: "active+clean" 2021-07-26T06:37:22.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2021-07-26T06:37:22.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2021-07-26T06:37:22.622 INFO:tasks.workunit.client.0.smithi071.stdout:"active+clean" 2021-07-26T06:37:22.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2021-07-26T06:37:22.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 2 == 4 ']' 2021-07-26T06:37:22.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2021-07-26T06:37:22.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2021-07-26T06:37:22.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2021-07-26T06:37:22.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2021-07-26T06:37:22.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2021-07-26T06:37:23.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2021-07-26T06:37:23.046 INFO:tasks.workunit.client.0.smithi071.stdout:3 ) state now: "active+clean" 2021-07-26T06:37:23.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean"' 2021-07-26T06:37:23.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2021-07-26T06:37:23.048 INFO:tasks.workunit.client.0.smithi071.stdout:"active+clean" 2021-07-26T06:37:23.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2021-07-26T06:37:23.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 3 == 4 ']' 2021-07-26T06:37:23.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2021-07-26T06:37:23.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2021-07-26T06:37:23.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2021-07-26T06:37:23.348 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2021-07-26T06:37:23.348 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2021-07-26T06:37:23.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2021-07-26T06:37:23.470 INFO:tasks.workunit.client.0.smithi071.stdout:4 ) state now: "active+clean" 2021-07-26T06:37:23.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 4 ') state now: ' '"active+clean"' 2021-07-26T06:37:23.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2021-07-26T06:37:23.472 INFO:tasks.workunit.client.0.smithi071.stdout:"active+clean" 2021-07-26T06:37:23.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2021-07-26T06:37:23.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 4 == 4 ']' 2021-07-26T06:37:23.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:354: initiate_and_fetch_state: echo 'loop --------> ' 4 2021-07-26T06:37:23.473 INFO:tasks.workunit.client.0.smithi071.stdout:loop --------> 4 2021-07-26T06:37:23.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2021-07-26T06:37:23.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2021-07-26T06:37:23.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2021-07-26T06:37:23.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2021-07-26T06:37:23.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2021-07-26T06:37:23.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2021-07-26T06:37:23.894 INFO:tasks.workunit.client.0.smithi071.stdout:5 ) state now: "active+clean" 2021-07-26T06:37:23.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 5 ') state now: ' '"active+clean"' 2021-07-26T06:37:23.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2021-07-26T06:37:23.895 INFO:tasks.workunit.client.0.smithi071.stdout:"active+clean" 2021-07-26T06:37:23.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2021-07-26T06:37:23.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 5 == 4 ']' 2021-07-26T06:37:23.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2021-07-26T06:37:24.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2021-07-26T06:37:24.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2021-07-26T06:37:24.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2021-07-26T06:37:24.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2021-07-26T06:37:24.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2021-07-26T06:37:24.318 INFO:tasks.workunit.client.0.smithi071.stdout:6 ) state now: "active+clean" 2021-07-26T06:37:24.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 6 ') state now: ' '"active+clean"' 2021-07-26T06:37:24.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2021-07-26T06:37:24.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2021-07-26T06:37:24.320 INFO:tasks.workunit.client.0.smithi071.stdout:"active+clean" 2021-07-26T06:37:24.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 6 == 4 ']' 2021-07-26T06:37:24.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2021-07-26T06:37:24.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2021-07-26T06:37:24.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2021-07-26T06:37:24.620 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2021-07-26T06:37:24.620 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2021-07-26T06:37:24.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2021-07-26T06:37:24.756 INFO:tasks.workunit.client.0.smithi071.stdout:7 ) state now: "active+clean" 2021-07-26T06:37:24.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 7 ') state now: ' '"active+clean"' 2021-07-26T06:37:24.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2021-07-26T06:37:24.757 INFO:tasks.workunit.client.0.smithi071.stdout:"active+clean" 2021-07-26T06:37:24.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2021-07-26T06:37:24.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 7 == 4 ']' 2021-07-26T06:37:24.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2021-07-26T06:37:25.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2021-07-26T06:37:25.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2021-07-26T06:37:25.057 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2021-07-26T06:37:25.058 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2021-07-26T06:37:25.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2021-07-26T06:37:25.178 INFO:tasks.workunit.client.0.smithi071.stdout:8 ) state now: "active+clean" 2021-07-26T06:37:25.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 8 ') state now: ' '"active+clean"' 2021-07-26T06:37:25.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2021-07-26T06:37:25.179 INFO:tasks.workunit.client.0.smithi071.stdout:"active+clean" 2021-07-26T06:37:25.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2021-07-26T06:37:25.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 8 == 4 ']' 2021-07-26T06:37:25.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2021-07-26T06:37:25.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2021-07-26T06:37:25.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2021-07-26T06:37:25.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2021-07-26T06:37:25.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2021-07-26T06:37:25.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2021-07-26T06:37:25.603 INFO:tasks.workunit.client.0.smithi071.stdout:9 ) state now: "active+clean" 2021-07-26T06:37:25.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 9 ') state now: ' '"active+clean"' 2021-07-26T06:37:25.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2021-07-26T06:37:25.604 INFO:tasks.workunit.client.0.smithi071.stdout:"active+clean" 2021-07-26T06:37:25.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2021-07-26T06:37:25.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 9 == 4 ']' 2021-07-26T06:37:25.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2021-07-26T06:37:25.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2021-07-26T06:37:25.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2021-07-26T06:37:25.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2021-07-26T06:37:25.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2021-07-26T06:37:26.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2021-07-26T06:37:26.027 INFO:tasks.workunit.client.0.smithi071.stdout:10 ) state now: "active+clean" 2021-07-26T06:37:26.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 10 ') state now: ' '"active+clean"' 2021-07-26T06:37:26.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2021-07-26T06:37:26.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2021-07-26T06:37:26.030 INFO:tasks.workunit.client.0.smithi071.stdout:"active+clean" 2021-07-26T06:37:26.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 0 == 4 ']' 2021-07-26T06:37:26.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2021-07-26T06:37:26.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2021-07-26T06:37:26.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2021-07-26T06:37:26.329 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2021-07-26T06:37:26.329 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2021-07-26T06:37:26.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2021-07-26T06:37:26.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 11 ') state now: ' '"active+clean"' 2021-07-26T06:37:26.451 INFO:tasks.workunit.client.0.smithi071.stdout:11 ) state now: "active+clean" 2021-07-26T06:37:26.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2021-07-26T06:37:26.452 INFO:tasks.workunit.client.0.smithi071.stdout:"active+clean" 2021-07-26T06:37:26.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2021-07-26T06:37:26.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 1 == 4 ']' 2021-07-26T06:37:26.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2021-07-26T06:37:26.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2021-07-26T06:37:26.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2021-07-26T06:37:26.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2021-07-26T06:37:26.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2021-07-26T06:37:26.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2021-07-26T06:37:26.875 INFO:tasks.workunit.client.0.smithi071.stdout:12 ) state now: "active+clean+scrubbing+deep" 2021-07-26T06:37:26.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 12 ') state now: ' '"active+clean+scrubbing+deep"' 2021-07-26T06:37:26.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2021-07-26T06:37:26.877 INFO:tasks.workunit.client.0.smithi071.stdout:found scrub 2021-07-26T06:37:26.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:347: initiate_and_fetch_state: echo 'found scrub' 2021-07-26T06:37:26.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:347: initiate_and_fetch_state: return 0 2021-07-26T06:37:26.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2021-07-26T06:37:26.879 INFO:tasks.workunit.client.0.smithi071.stdout:initiate_and_fetch_state ret: 0 2021-07-26T06:37:26.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2021-07-26T06:37:26.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2021-07-26T06:37:26.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local daemon=osd 2021-07-26T06:37:26.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: set_config: local id=td/osd-scrub-repair 2021-07-26T06:37:26.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: local config=osd_scrub_sleep 2021-07-26T06:37:26.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: local value=0 2021-07-26T06:37:26.882 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: jq 'has("success")' 2021-07-26T06:37:26.883 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: get_asok_path osd.td/osd-scrub-repair 2021-07-26T06:37:26.883 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2021-07-26T06:37:26.883 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2021-07-26T06:37:26.883 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:37:26.884 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:26.884 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:26.884 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.td/osd-scrub-repair.asok 2021-07-26T06:37:26.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.128899/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2021-07-26T06:37:26.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: test == true 2021-07-26T06:37:26.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1110: test: ==: unary operator expected 2021-07-26T06:37:26.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2021-07-26T06:37:26.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2021-07-26T06:37:26.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: local the_osd=osd.1 2021-07-26T06:37:26.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: local pgid=1.0 2021-07-26T06:37:26.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i=0 )) 2021-07-26T06:37:26.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i < 40 )) 2021-07-26T06:37:26.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: ceph pg 1.0 query --format json 2021-07-26T06:37:26.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: jq .state 2021-07-26T06:37:27.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2021-07-26T06:37:27.080 INFO:tasks.workunit.client.0.smithi071.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2021-07-26T06:37:27.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2021-07-26T06:37:27.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2021-07-26T06:37:27.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:371: wait_end_of_scrub: '[' 0 == 4 ']' 2021-07-26T06:37:27.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: wait_end_of_scrub: sleep 0.3 2021-07-26T06:37:27.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i++ )) 2021-07-26T06:37:27.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i < 40 )) 2021-07-26T06:37:27.382 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: ceph pg 1.0 query --format json 2021-07-26T06:37:27.383 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: jq .state 2021-07-26T06:37:27.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2021-07-26T06:37:27.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2021-07-26T06:37:27.504 INFO:tasks.workunit.client.0.smithi071.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2021-07-26T06:37:27.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2021-07-26T06:37:27.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:371: wait_end_of_scrub: '[' 1 == 4 ']' 2021-07-26T06:37:27.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: wait_end_of_scrub: sleep 0.3 2021-07-26T06:37:27.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i++ )) 2021-07-26T06:37:27.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i < 40 )) 2021-07-26T06:37:27.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: ceph pg 1.0 query --format json 2021-07-26T06:37:27.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: jq .state 2021-07-26T06:37:27.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2021-07-26T06:37:27.927 INFO:tasks.workunit.client.0.smithi071.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2021-07-26T06:37:27.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2021-07-26T06:37:27.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2021-07-26T06:37:27.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:371: wait_end_of_scrub: '[' 2 == 4 ']' 2021-07-26T06:37:27.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: wait_end_of_scrub: sleep 0.3 2021-07-26T06:37:28.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i++ )) 2021-07-26T06:37:28.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i < 40 )) 2021-07-26T06:37:28.229 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: ceph pg 1.0 query --format json 2021-07-26T06:37:28.229 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: jq .state 2021-07-26T06:37:28.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2021-07-26T06:37:28.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2021-07-26T06:37:28.350 INFO:tasks.workunit.client.0.smithi071.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2021-07-26T06:37:28.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2021-07-26T06:37:28.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:371: wait_end_of_scrub: '[' 3 == 4 ']' 2021-07-26T06:37:28.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: wait_end_of_scrub: sleep 0.3 2021-07-26T06:37:28.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i++ )) 2021-07-26T06:37:28.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i < 40 )) 2021-07-26T06:37:28.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: ceph pg 1.0 query --format json 2021-07-26T06:37:28.652 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: jq .state 2021-07-26T06:37:28.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2021-07-26T06:37:28.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2021-07-26T06:37:28.817 INFO:tasks.workunit.client.0.smithi071.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2021-07-26T06:37:28.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2021-07-26T06:37:28.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:371: wait_end_of_scrub: '[' 4 == 4 ']' 2021-07-26T06:37:28.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:372: wait_end_of_scrub: flush_pg_stats 2021-07-26T06:37:28.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:37:28.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:37:29.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:37:29.082 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:37:29.082 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:37:29.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:37:29.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:29.083 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:37:29.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182407 2021-07-26T06:37:29.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182407 2021-07-26T06:37:29.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182407' 2021-07-26T06:37:29.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:29.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:37:29.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672973 2021-07-26T06:37:29.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672973 2021-07-26T06:37:29.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182407 1-42949672973' 2021-07-26T06:37:29.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:29.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:37:29.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509451 2021-07-26T06:37:29.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509451 2021-07-26T06:37:29.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-107374182407 1-42949672973 2-64424509451' 2021-07-26T06:37:29.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:29.438 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-107374182407 2021-07-26T06:37:29.438 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:29.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:37:29.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-107374182407 2021-07-26T06:37:29.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:29.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182407 2021-07-26T06:37:29.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2021-07-26T06:37:29.440 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 107374182407 2021-07-26T06:37:29.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:37:29.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182406 -lt 107374182407 2021-07-26T06:37:29.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:37:30.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:37:30.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:37:30.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182407 -lt 107374182407 2021-07-26T06:37:30.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:30.985 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672973 2021-07-26T06:37:30.986 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:30.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:37:30.987 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672973 2021-07-26T06:37:30.987 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:30.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672973 2021-07-26T06:37:30.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2021-07-26T06:37:30.989 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672973 2021-07-26T06:37:30.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:37:31.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672973 -lt 42949672973 2021-07-26T06:37:31.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:31.275 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509451 2021-07-26T06:37:31.275 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:31.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:37:31.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509451 2021-07-26T06:37:31.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:31.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509451 2021-07-26T06:37:31.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2021-07-26T06:37:31.278 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509451 2021-07-26T06:37:31.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:37:31.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509452 -lt 64424509451 2021-07-26T06:37:31.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: wait_end_of_scrub: sleep 0.3 2021-07-26T06:37:31.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i++ )) 2021-07-26T06:37:31.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i < 40 )) 2021-07-26T06:37:31.893 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: ceph pg 1.0 query --format json 2021-07-26T06:37:31.893 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: jq .state 2021-07-26T06:37:32.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: st='"active+clean"' 2021-07-26T06:37:32.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2021-07-26T06:37:32.015 INFO:tasks.workunit.client.0.smithi071.stdout:wait-scrub-end state now: "active+clean" 2021-07-26T06:37:32.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2021-07-26T06:37:32.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: break 2021-07-26T06:37:32.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:377: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2021-07-26T06:37:32.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:382: wait_end_of_scrub: return 0 2021-07-26T06:37:32.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2021-07-26T06:37:32.263 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:37:32.263 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 1 0 0 0 0 7 0 0 1 1 active+clean+scrubbing+deep 2021-07-26T06:37:26.799486+0000 21'1 26:43 [1,0] 1 [1,0] 1 0'0 2021-07-19T06:35:41.644667+0000 0'0 2021-07-19T06:35:41.536972+0000 0 2021-07-26T06:37:32.264 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T06:37:32.264 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:37:32.265 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T06:37:32.275 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2021-07-26T06:37:32.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T06:37:32.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:37:32.276 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool SOMETHING 2021-07-26T06:37:32.276 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:37:32.277 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:37:32.277 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:37:32.277 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:37:32.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:37:32.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:37:32.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:37:32.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2021-07-26T06:37:32.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:37:32.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:37:32.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:37:32.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:37:32.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2021-07-26T06:37:32.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:37:32.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:37:32.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:37:32.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:37:32.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:37:32.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:37:32.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:37:32.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:37:32.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:37:32.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:37:32.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:37:32.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2021-07-26T06:37:32.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:37:32.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:37:32.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:37:32.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:37:32.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:37:32.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2021-07-26T06:37:34.350 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:37:34.351 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:37:34.351 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:37:34.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:37:34.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:37:34.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:37:34.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:37:34.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:37:34.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:37:34.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:37:34.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:37:34.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:37:34.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:37:34.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:37:34.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:37:34.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:37:34.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:37:34.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:37:34.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:37:34.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:37:34.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:37:34.630 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:37:34.630 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:37:34.630 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:34.630 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:34.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:37:34.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:37:34.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:37:34.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:37:34.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:37:34.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:37:34.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:37:34.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:37:34.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:37:34.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:37:34.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:37:34.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:37:34.634 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:37:34.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:37:34.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:37:34.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:37:34.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:37:34.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:37:34.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:37:34.649 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:34.651+0000 7f365446b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:34.651 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:34.653+0000 7f365446b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:34.652 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:34.654+0000 7f365446b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:34.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:37:34.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:37:34.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:37:34.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:37:34.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:37:34.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:34.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:37:34.905 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:37:34.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:34.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:35.175 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:35.177+0000 7f365446b0c0 -1 Falling back to public interface 2021-07-26T06:37:35.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:36.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:36.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:36.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:37:36.218 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:37:36.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:36.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:36.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:37.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:37.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:37.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:37:37.498 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:37:37.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:37.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:37.643 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:37.645+0000 7f365446b0c0 -1 osd.0 26 log_to_monitors {default=true} 2021-07-26T06:37:37.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:38.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:38.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:38.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:37:38.832 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:37:38.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:38.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:39.109 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/186853,v1:127.0.0.1:6803/186853] [v2:127.0.0.1:6804/186853,v1:127.0.0.1:6805/186853] exists,up 8c00e533-4d59-4dfe-9871-4461f609206d 2021-07-26T06:37:39.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:37:39.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:37:39.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:37:39.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:37:39.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:37:39.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:37:39.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:37:39.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:37:39.112 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:37:39.112 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:37:39.112 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:37:39.112 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:37:39.113 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:37:39.113 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:37:39.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:37:39.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:37:39.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:37:39.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:37:39.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:37:39.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:37:39.468 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:37:39.468 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:37:39.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:37:39.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:39.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:37:39.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018882 2021-07-26T06:37:39.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018882 2021-07-26T06:37:39.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018882' 2021-07-26T06:37:39.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:39.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:37:39.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672976 2021-07-26T06:37:39.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672976 2021-07-26T06:37:39.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018882 1-42949672976' 2021-07-26T06:37:39.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:39.707 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:37:39.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509454 2021-07-26T06:37:39.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509454 2021-07-26T06:37:39.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-128849018882 1-42949672976 2-64424509454' 2021-07-26T06:37:39.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:39.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-128849018882 2021-07-26T06:37:39.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:39.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:37:39.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-128849018882 2021-07-26T06:37:39.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:39.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018882 2021-07-26T06:37:39.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2021-07-26T06:37:39.828 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 128849018882 2021-07-26T06:37:39.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:37:40.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 128849018882 2021-07-26T06:37:40.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:37:41.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:37:41.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:37:41.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018882 -lt 128849018882 2021-07-26T06:37:41.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:41.373 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672976 2021-07-26T06:37:41.373 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:41.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:37:41.374 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672976 2021-07-26T06:37:41.375 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:41.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672976 2021-07-26T06:37:41.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2021-07-26T06:37:41.376 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672976 2021-07-26T06:37:41.376 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:37:41.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672976 -lt 42949672976 2021-07-26T06:37:41.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:41.647 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509454 2021-07-26T06:37:41.647 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:41.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:37:41.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509454 2021-07-26T06:37:41.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:41.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509454 2021-07-26T06:37:41.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2021-07-26T06:37:41.649 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509454 2021-07-26T06:37:41.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:37:41.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509455 -lt 64424509454 2021-07-26T06:37:41.957 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:37:41.957 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:37:41.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:37:42.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:37:42.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:37:42.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:37:42.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:37:42.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:37:42.319 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:37:42.319 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:37:42.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:37:42.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:37:42.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:37:42.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:37:42.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:37:42.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:37:42.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:37:42.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:37:42.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2021-07-26T06:37:42.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T06:37:42.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:37:42.967 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool SOMETHING 2021-07-26T06:37:42.967 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T06:37:42.967 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:37:42.967 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:37:42.967 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:37:43.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:37:43.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool SOMETHING 2021-07-26T06:37:43.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:37:43.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2021-07-26T06:37:43.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:37:43.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:37:43.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:37:43.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:37:43.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2021-07-26T06:37:43.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:37:43.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:37:43.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:37:43.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:37:43.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:37:43.509 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:37:43.509 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:37:43.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:37:43.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:37:43.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:37:43.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:37:43.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2021-07-26T06:37:43.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:37:43.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:37:43.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:37:43.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:37:43.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:37:43.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2021-07-26T06:37:45.286 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:37:45.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:37:45.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:37:45.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:37:45.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:37:45.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:37:45.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:37:45.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:37:45.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:37:45.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:37:45.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:37:45.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:37:45.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:37:45.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:37:45.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:37:45.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:37:45.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:37:45.831 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:37:45.831 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:37:45.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:37:45.832 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:37:45.832 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:45.832 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:45.833 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:37:45.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:37:45.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:37:45.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:37:45.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:37:45.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:37:45.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:37:45.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:37:45.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:37:45.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:37:45.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:37:45.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:37:45.835 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:37:45.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:37:45.836 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:37:45.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:37:45.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:37:45.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:37:45.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:37:45.853 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:45.855+0000 7f12fa98f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:45.855 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:45.857+0000 7f12fa98f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:45.856 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:45.858+0000 7f12fa98f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:46.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:37:46.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:37:46.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:37:46.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:37:46.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:37:46.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:46.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:37:46.109 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:37:46.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:46.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:46.373 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:46.374+0000 7f12fa98f0c0 -1 Falling back to public interface 2021-07-26T06:37:46.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:47.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:47.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:47.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:37:47.452 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:37:47.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:47.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:47.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:48.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:48.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:48.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:37:48.725 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:37:48.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:48.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:48.892 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:48.893+0000 7f12fa98f0c0 -1 osd.0 31 log_to_monitors {default=true} 2021-07-26T06:37:49.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:50.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:50.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:50.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:37:50.049 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:37:50.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:50.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:50.326 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/188251,v1:127.0.0.1:6803/188251] [v2:127.0.0.1:6804/188251,v1:127.0.0.1:6805/188251] exists,up 8c00e533-4d59-4dfe-9871-4461f609206d 2021-07-26T06:37:50.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:37:50.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:37:50.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:37:50.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:37:50.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:37:50.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:37:50.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:37:50.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:37:50.328 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:37:50.328 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:37:50.328 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:37:50.329 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:37:50.329 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:37:50.329 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:37:50.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:37:50.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:37:50.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:37:50.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:37:50.419 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:37:50.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:37:50.697 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:37:50.697 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:37:50.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:37:50.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:50.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:37:50.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855362 2021-07-26T06:37:50.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855362 2021-07-26T06:37:50.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-150323855362' 2021-07-26T06:37:50.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:50.848 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:37:50.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672979 2021-07-26T06:37:50.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672979 2021-07-26T06:37:50.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-150323855362 1-42949672979' 2021-07-26T06:37:50.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:37:50.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:37:51.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509458 2021-07-26T06:37:51.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509458 2021-07-26T06:37:51.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-150323855362 1-42949672979 2-64424509458' 2021-07-26T06:37:51.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:51.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-150323855362 2021-07-26T06:37:51.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:51.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:37:51.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-150323855362 2021-07-26T06:37:51.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:51.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855362 2021-07-26T06:37:51.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2021-07-26T06:37:51.088 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 150323855362 2021-07-26T06:37:51.088 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:37:51.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 150323855362 2021-07-26T06:37:51.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:37:52.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:37:52.354 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:37:52.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855362 -lt 150323855362 2021-07-26T06:37:52.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:52.628 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672979 2021-07-26T06:37:52.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:52.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:37:52.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672979 2021-07-26T06:37:52.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:52.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672979 2021-07-26T06:37:52.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2021-07-26T06:37:52.632 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672979 2021-07-26T06:37:52.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:37:52.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672979 -lt 42949672979 2021-07-26T06:37:52.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:37:52.930 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509458 2021-07-26T06:37:52.930 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:37:52.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:37:52.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509458 2021-07-26T06:37:52.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:37:52.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509458 2021-07-26T06:37:52.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2021-07-26T06:37:52.933 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509458 2021-07-26T06:37:52.933 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:37:53.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509458 -lt 64424509458 2021-07-26T06:37:53.198 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:37:53.199 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:37:53.199 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:37:53.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:37:53.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:37:53.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:37:53.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:37:53.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:37:53.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:37:53.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:37:53.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:37:53.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:37:53.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:37:53.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:37:53.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:37:54.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:37:54.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:37:54.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:37:54.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:37:54.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2021-07-26T06:37:54.183 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:37:29.806+0000 7f73d5474700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber pg(1.0) scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2021-07-26T06:37:54.183 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:37:29.806+0000 7f73d5474700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] PrimaryLog scrubber pg(1.0) _scrub_finish info stats: valid 2021-07-26T06:37:54.184 INFO:tasks.workunit.client.0.smithi071.stdout:2021-07-26T06:37:29.806+0000 7f73d5474700 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber pg(1.0) scrub_finish All may be fixed 2021-07-26T06:37:54.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:37:54.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:37:54.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:37:54.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:37:54.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:37:54.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:37:54.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:37:54.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:37:54.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:37:54.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:37:54.292 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:37:54.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:37:54.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:37:54.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:37:54.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:37:54.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:37:54.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:37:54.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:37:54.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:37:54.296 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:37:54.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:37:54.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:37:54.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:37:54.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:37:54.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:54.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:54.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:37:54.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:37:54.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:37:54.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:37:54.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:37:54.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:37:54.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:37:54.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:37:54.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:37:54.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:37:54.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:37:54.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:37:54.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:37:54.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:37:54.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:37:54.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:37:54.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:37:54.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:37:54.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:37:54.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:37:54.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:37:54.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:37:54.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:37:54.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:37:54.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:37:54.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:37:54.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:37:54.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:37:54.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:37:54.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:37:54.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:54.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:54.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:37:54.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:37:54.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:37:54.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:37:54.321 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:37:54.321 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:54.321 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:54.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:37:54.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:37:54.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:37:54.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:37:54.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:37:54.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2021-07-26T06:37:54.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:312: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2021-07-26T06:37:54.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:267: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2021-07-26T06:37:54.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local allow_overwrites=false 2021-07-26T06:37:54.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local poolname=ecpool 2021-07-26T06:37:54.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2021-07-26T06:37:54.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:37:54.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:37:54.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:37:54.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:37:54.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:37:54.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T06:37:54.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:37:54.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:37:54.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:37:54.723 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:37:54.723 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:54.723 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:54.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:37:54.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:37:55.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:37:55.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:37:55.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:37:55.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:37:55.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:37:55.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:37:55.026 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:37:55.026 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:37:55.026 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:37:55.027 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:37:55.027 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:55.027 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:55.027 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:37:55.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:37:55.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:37:55.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:37:55.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:37:55.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:37:55.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:37:55.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:37:55.123 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:37:55.124 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:37:55.124 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:37:55.124 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:37:55.124 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:55.125 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:55.125 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:37:55.125 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:37:55.125 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:37:55.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2021-07-26T06:37:55.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:37:55.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:37:55.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:37:55.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:37:55.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:37:55.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:37:56.060 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:37:56.061 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:37:56.061 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:37:56.061 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:37:56.061 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:56.062 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:56.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:37:56.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:37:56.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:37:56.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:278: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:37:56.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: seq 0 2 2021-07-26T06:37:56.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: for id in $(seq 0 2) 2021-07-26T06:37:56.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: '[' false = true ']' 2021-07-26T06:37:56.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: run_osd_filestore td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:37:56.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:37:56.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:37:56.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2021-07-26T06:37:56.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:37:56.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:37:56.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:37:56.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:37:56.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:37:56.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:37:56.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:37:56.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:37:56.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:37:56.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:37:56.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:37:56.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:37:56.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:37:56.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:37:56.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:37:56.095 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:37:56.096 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:56.096 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:56.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:37:56.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:37:56.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:37:56.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:37:56.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:37:56.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:37:56.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:37:56.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:37:56.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:37:56.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:37:56.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:37:56.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:37:56.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:37:56.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:37:56.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=5a7b0301-03bf-42ff-adc6-248aee2e3901 2021-07-26T06:37:56.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd2 5a7b0301-03bf-42ff-adc6-248aee2e3901' 2021-07-26T06:37:56.101 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 5a7b0301-03bf-42ff-adc6-248aee2e3901 2021-07-26T06:37:56.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:37:56.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQBEWP5gR5RHBhAANQN7nfzt6tn5aOF4GIRTDQ== 2021-07-26T06:37:56.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQBEWP5gR5RHBhAANQN7nfzt6tn5aOF4GIRTDQ=="}' 2021-07-26T06:37:56.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 5a7b0301-03bf-42ff-adc6-248aee2e3901 -i td/osd-scrub-repair/0/new.json 2021-07-26T06:37:56.366 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:37:56.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:37:56.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBEWP5gR5RHBhAANQN7nfzt6tn5aOF4GIRTDQ== --osd-uuid 5a7b0301-03bf-42ff-adc6-248aee2e3901 --osd-objectstore=filestore 2021-07-26T06:37:56.405 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:56.405+0000 7f13cb6ca0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:56.407 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:56.407+0000 7f13cb6ca0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:56.407 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:56.408+0000 7f13cb6ca0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:56.960 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:56.961+0000 7f13cb6ca0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:37:57.464 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:57.464+0000 7f13cb6ca0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:37:57.465 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:57.464+0000 7f13cb6ca0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:37:57.465 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:57.464+0000 7f13cb6ca0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:37:57.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:37:57.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:37:57.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd0 key to auth repository 2021-07-26T06:37:57.969 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:37:57.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:37:58.146 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2021-07-26T06:37:58.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.0 2021-07-26T06:37:58.147 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:37:58.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:37:58.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:37:58.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:37:58.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:37:58.173 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:58.173+0000 7f445d5ec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:58.175 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:58.175+0000 7f445d5ec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:58.176 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:58.176+0000 7f445d5ec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:58.178 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:58.178+0000 7f445d5ec0c0 -1 Falling back to public interface 2021-07-26T06:37:58.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 0 2021-07-26T06:37:58.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:37:58.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:37:58.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:37:58.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:37:58.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:58.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:37:58.535 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:37:58.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:58.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:58.737 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:58.738+0000 7f445d5ec0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:37:58.745 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:58.746+0000 7f445d5ec0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:37:58.746 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:58.746+0000 7f445d5ec0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:37:58.751 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:37:58.752+0000 7f445d5ec0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:37:58.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:59.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:59.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:59.768 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:37:59.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:37:59.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:59.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:59.943 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/189540,v1:127.0.0.1:6801/189540] [v2:127.0.0.1:6802/189540,v1:127.0.0.1:6803/189540] exists,up 5a7b0301-03bf-42ff-adc6-248aee2e3901 2021-07-26T06:37:59.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:37:59.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:37:59.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:37:59.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: for id in $(seq 0 2) 2021-07-26T06:37:59.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: '[' false = true ']' 2021-07-26T06:37:59.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: run_osd_filestore td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:37:59.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:37:59.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:37:59.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2021-07-26T06:37:59.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:37:59.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:37:59.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:37:59.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:37:59.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:37:59.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:37:59.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:37:59.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:37:59.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:37:59.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:37:59.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:37:59.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:37:59.954 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:37:59.954 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:37:59.954 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:37:59.954 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:37:59.954 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:59.955 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:37:59.955 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:37:59.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:37:59.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:37:59.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:37:59.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:37:59.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:37:59.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:37:59.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:37:59.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:37:59.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:37:59.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:37:59.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:37:59.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:37:59.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:37:59.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=73c31ab6-d046-4604-9d36-67e7f791f953 2021-07-26T06:37:59.958 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 73c31ab6-d046-4604-9d36-67e7f791f953 2021-07-26T06:37:59.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd2 73c31ab6-d046-4604-9d36-67e7f791f953' 2021-07-26T06:37:59.959 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:37:59.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQBHWP5gsdFoORAAmGpa4Faeui4xxXRqZn52CQ== 2021-07-26T06:37:59.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQBHWP5gsdFoORAAmGpa4Faeui4xxXRqZn52CQ=="}' 2021-07-26T06:37:59.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 73c31ab6-d046-4604-9d36-67e7f791f953 -i td/osd-scrub-repair/1/new.json 2021-07-26T06:38:00.278 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:38:00.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:38:00.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBHWP5gsdFoORAAmGpa4Faeui4xxXRqZn52CQ== --osd-uuid 73c31ab6-d046-4604-9d36-67e7f791f953 --osd-objectstore=filestore 2021-07-26T06:38:00.318 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:00.318+0000 7f662a9e20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:00.328 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:00.328+0000 7f662a9e20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:00.329 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:00.329+0000 7f662a9e20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:01.100 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:01.100+0000 7f662a9e20c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:38:02.046 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:02.046+0000 7f662a9e20c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:38:02.086 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:02.087+0000 7f662a9e20c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:38:02.087 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:02.087+0000 7f662a9e20c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:38:02.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:38:02.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:38:02.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd1 key to auth repository 2021-07-26T06:38:02.714 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:38:02.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:38:03.056 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2021-07-26T06:38:03.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.1 2021-07-26T06:38:03.058 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:38:03.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:38:03.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:38:03.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:38:03.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:38:03.083 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:03.084+0000 7f3c180350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:03.085 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:03.085+0000 7f3c180350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:03.086 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:03.086+0000 7f3c180350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:03.087 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:03.087+0000 7f3c180350c0 -1 Falling back to public interface 2021-07-26T06:38:03.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 1 2021-07-26T06:38:03.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:38:03.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:38:03.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:38:03.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:38:03.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:03.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:38:03.479 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:38:03.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:03.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:38:03.638 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:03.638+0000 7f3c180350c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:38:03.670 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:03.671+0000 7f3c180350c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:38:03.671 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:03.671+0000 7f3c180350c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:38:03.676 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:03.676+0000 7f3c180350c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:38:03.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:38:04.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:38:04.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:04.748 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:38:04.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:38:04.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:04.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:38:05.023 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/189910,v1:127.0.0.1:6811/189910] [v2:127.0.0.1:6812/189910,v1:127.0.0.1:6813/189910] exists,up 73c31ab6-d046-4604-9d36-67e7f791f953 2021-07-26T06:38:05.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:38:05.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:38:05.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:38:05.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: for id in $(seq 0 2) 2021-07-26T06:38:05.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: '[' false = true ']' 2021-07-26T06:38:05.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: run_osd_filestore td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:38:05.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:38:05.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:38:05.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2021-07-26T06:38:05.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:38:05.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:38:05.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:38:05.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:38:05.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:38:05.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:38:05.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:38:05.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:38:05.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:38:05.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:38:05.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:38:05.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:38:05.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:38:05.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:38:05.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:38:05.034 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:38:05.035 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:05.035 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:38:05.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:38:05.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:38:05.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:38:05.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:38:05.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:38:05.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:38:05.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:38:05.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:38:05.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:38:05.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:38:05.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:38:05.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:38:05.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:38:05.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:38:05.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=af54be29-6849-44d7-b343-4b3e2725dada 2021-07-26T06:38:05.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd2 af54be29-6849-44d7-b343-4b3e2725dada' 2021-07-26T06:38:05.039 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 af54be29-6849-44d7-b343-4b3e2725dada 2021-07-26T06:38:05.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:38:05.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQBNWP5gqE6XAhAAhirdYE0u9Jfuj73uSCZu8w== 2021-07-26T06:38:05.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQBNWP5gqE6XAhAAhirdYE0u9Jfuj73uSCZu8w=="}' 2021-07-26T06:38:05.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new af54be29-6849-44d7-b343-4b3e2725dada -i td/osd-scrub-repair/2/new.json 2021-07-26T06:38:05.435 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:38:05.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2021-07-26T06:38:05.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBNWP5gqE6XAhAAhirdYE0u9Jfuj73uSCZu8w== --osd-uuid af54be29-6849-44d7-b343-4b3e2725dada --osd-objectstore=filestore 2021-07-26T06:38:05.476 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:05.476+0000 7f116984a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:05.477 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:05.478+0000 7f116984a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:05.478 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:05.478+0000 7f116984a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:06.148 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:06.149+0000 7f116984a0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:38:06.677 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:06.677+0000 7f116984a0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:38:06.677 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:06.677+0000 7f116984a0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:38:06.677 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:06.677+0000 7f116984a0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:38:07.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T06:38:07.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:38:07.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd2 key to auth repository 2021-07-26T06:38:07.498 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:38:07.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:38:07.838 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:38:07.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.2 2021-07-26T06:38:07.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:38:07.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:38:07.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:38:07.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:38:07.863 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:07.864+0000 7f5cfb6460c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:07.865 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:07.865+0000 7f5cfb6460c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:07.866 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:07.866+0000 7f5cfb6460c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:07.867 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:07.867+0000 7f5cfb6460c0 -1 Falling back to public interface 2021-07-26T06:38:08.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 2 2021-07-26T06:38:08.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:38:08.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:38:08.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:38:08.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:38:08.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:08.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:38:08.258 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:38:08.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:08.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:38:08.553 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:08.553+0000 7f5cfb6460c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:38:08.554 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:08.554+0000 7f5cfb6460c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:38:08.554 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:08.554+0000 7f5cfb6460c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:38:08.559 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:08.560+0000 7f5cfb6460c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:38:08.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:38:09.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:38:09.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:09.660 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:38:09.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:38:09.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:09.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:38:09.973 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/190194,v1:127.0.0.1:6819/190194] [v2:127.0.0.1:6820/190194,v1:127.0.0.1:6821/190194] exists,up af54be29-6849-44d7-b343-4b3e2725dada 2021-07-26T06:38:09.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:38:09.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:38:09.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:38:09.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: auto_repair_erasure_coded: create_rbd_pool 2021-07-26T06:38:09.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:38:10.234 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T06:38:10.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:38:10.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:38:10.650 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T06:38:10.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:38:11.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:38:13.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: wait_for_clean 2021-07-26T06:38:13.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:38:13.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:38:13.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:38:13.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:38:13.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:38:13.635 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:38:13.635 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:38:13.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:38:13.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:38:13.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:38:13.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:38:13.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:38:13.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:38:13.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:38:13.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:38:14.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:38:14.121 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:38:14.121 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:38:14.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:38:14.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:14.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:38:14.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836484 2021-07-26T06:38:14.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836484 2021-07-26T06:38:14.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484' 2021-07-26T06:38:14.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:14.239 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:38:14.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672963 2021-07-26T06:38:14.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672963 2021-07-26T06:38:14.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484 1-42949672963' 2021-07-26T06:38:14.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:14.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:38:14.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=68719476738 2021-07-26T06:38:14.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 68719476738 2021-07-26T06:38:14.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-68719476738' 2021-07-26T06:38:14.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:14.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836484 2021-07-26T06:38:14.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:14.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:38:14.519 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836484 2021-07-26T06:38:14.520 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:14.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836484 2021-07-26T06:38:14.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2021-07-26T06:38:14.521 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836484 2021-07-26T06:38:14.521 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:38:14.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836483 -lt 21474836484 2021-07-26T06:38:14.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:38:15.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:38:15.838 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:38:16.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836484 -lt 21474836484 2021-07-26T06:38:16.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:16.210 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672963 2021-07-26T06:38:16.210 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:16.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:38:16.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672963 2021-07-26T06:38:16.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:16.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672963 2021-07-26T06:38:16.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2021-07-26T06:38:16.212 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672963 2021-07-26T06:38:16.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:38:16.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672963 -lt 42949672963 2021-07-26T06:38:16.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:16.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-68719476738 2021-07-26T06:38:16.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:16.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:38:16.481 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-68719476738 2021-07-26T06:38:16.481 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:16.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=68719476738 2021-07-26T06:38:16.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 68719476738' 2021-07-26T06:38:16.483 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 68719476738 2021-07-26T06:38:16.483 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:38:16.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 68719476738 -lt 68719476738 2021-07-26T06:38:16.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:38:16.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:38:16.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:38:17.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:38:17.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:38:17.116 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:38:17.116 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:38:17.116 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:38:17.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:38:17.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:38:17.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:38:17.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:38:17.378 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:38:17.378 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:38:17.378 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:38:17.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:38:17.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:38:17.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:38:17.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2021-07-26T06:38:17.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=ecpool 2021-07-26T06:38:17.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T06:38:17.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=false 2021-07-26T06:38:17.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T06:38:17.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2021-07-26T06:38:18.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2021-07-26T06:38:18.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2021-07-26T06:38:18.847 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' created 2021-07-26T06:38:18.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:38:19.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' false = true ']' 2021-07-26T06:38:19.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T06:38:19.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:38:19.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:38:19.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:38:19.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:38:19.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:38:19.862 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:38:19.862 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:38:19.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:38:19.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:38:19.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:38:19.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:38:19.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:38:19.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:38:19.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:38:19.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:38:20.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:38:20.457 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:38:20.457 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:38:20.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:38:20.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:20.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:38:20.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836486 2021-07-26T06:38:20.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836486 2021-07-26T06:38:20.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836486' 2021-07-26T06:38:20.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:20.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:38:20.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672966 2021-07-26T06:38:20.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672966 2021-07-26T06:38:20.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2021-07-26T06:38:20.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:20.692 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:38:20.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=68719476741 2021-07-26T06:38:20.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 68719476741 2021-07-26T06:38:20.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-68719476741' 2021-07-26T06:38:20.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:20.833 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836486 2021-07-26T06:38:20.833 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:20.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:38:20.834 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836486 2021-07-26T06:38:20.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:20.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836486 2021-07-26T06:38:20.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2021-07-26T06:38:20.836 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836486 2021-07-26T06:38:20.836 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:38:21.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836486 2021-07-26T06:38:21.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:21.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672966 2021-07-26T06:38:21.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:21.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:38:21.222 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672966 2021-07-26T06:38:21.222 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:21.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672966 2021-07-26T06:38:21.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-07-26T06:38:21.224 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672966 2021-07-26T06:38:21.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:38:21.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672966 -lt 42949672966 2021-07-26T06:38:21.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:21.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-68719476741 2021-07-26T06:38:21.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:21.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:38:21.531 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-68719476741 2021-07-26T06:38:21.531 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:21.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=68719476741 2021-07-26T06:38:21.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 68719476741' 2021-07-26T06:38:21.532 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 68719476741 2021-07-26T06:38:21.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:38:21.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 68719476741 -lt 68719476741 2021-07-26T06:38:21.797 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:38:21.797 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:38:21.797 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:38:22.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:38:22.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:38:22.126 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:38:22.126 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:38:22.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:38:22.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:38:22.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:38:22.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:38:22.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:38:22.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:38:22.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:38:22.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:38:22.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:38:22.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:38:22.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:38:22.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T06:38:22.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: auto_repair_erasure_coded: local payload=ABCDEF 2021-07-26T06:38:22.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:294: auto_repair_erasure_coded: echo ABCDEF 2021-07-26T06:38:22.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:295: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T06:38:22.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2021-07-26T06:38:22.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=ecpool 2021-07-26T06:38:22.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:38:22.875 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary ecpool SOMETHING 2021-07-26T06:38:22.875 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=ecpool 2021-07-26T06:38:22.876 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:38:22.876 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:38:22.876 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:38:23.169 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=2 2021-07-26T06:38:23.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:38:23.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2021-07-26T06:38:23.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:38:23.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:38:23.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:38:23.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:38:23.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:38:23.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:38:23.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:38:23.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:38:23.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:38:23.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:38:23.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:38:23.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:38:23.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:38:23.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:38:23.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:38:23.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:38:23.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:38:23.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:38:23.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:38:23.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:38:23.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:38:23.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:38:23.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:38:23.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2021-07-26T06:38:24.528 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2021-07-26T06:38:24.528 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:38:24.529 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:38:24.529 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:38:24.529 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:38:24.529 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2021-07-26T06:38:24.530 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:38:24.530 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:38:25.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:38:25.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:38:25.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:38:25.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:38:25.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:38:25.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:38:25.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:38:25.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:38:25.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:38:25.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:38:25.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:38:25.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:38:25.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:38:25.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:38:25.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:38:25.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:38:25.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:38:25.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:38:25.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:38:25.122 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:38:25.122 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:25.122 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:38:25.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:38:25.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:38:25.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:38:25.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:38:25.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:38:25.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:38:25.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:38:25.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:38:25.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:38:25.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:38:25.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:38:25.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:38:25.125 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:38:25.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:38:25.126 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:38:25.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:38:25.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:38:25.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:38:25.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:38:25.144 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:25.146+0000 7f69252c60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:25.146 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:25.148+0000 7f69252c60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:25.147 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:25.149+0000 7f69252c60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:25.149 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:25.150+0000 7f69252c60c0 -1 Falling back to public interface 2021-07-26T06:38:25.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:38:25.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:38:25.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:38:25.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:38:25.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:38:25.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:25.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:38:25.409 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:38:25.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:25.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:38:25.797 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:25.798+0000 7f69252c60c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:38:25.806 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:25.807+0000 7f69252c60c0 -1 journal do_read_entry(401408): bad header magic 2021-07-26T06:38:25.807 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:25.807+0000 7f69252c60c0 -1 journal do_read_entry(401408): bad header magic 2021-07-26T06:38:25.814 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:25.815+0000 7f69252c60c0 -1 osd.1 36 log_to_monitors {default=true} 2021-07-26T06:38:25.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:38:26.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:38:26.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:26.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:38:26.863 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:38:26.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:26.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:38:27.287 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/191349,v1:127.0.0.1:6811/191349] [v2:127.0.0.1:6812/191349,v1:127.0.0.1:6813/191349] exists,up 73c31ab6-d046-4604-9d36-67e7f791f953 2021-07-26T06:38:27.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:38:27.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:38:27.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:38:27.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:38:27.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:38:27.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:38:27.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:38:27.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:38:27.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:38:27.290 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:38:27.290 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:38:27.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:38:27.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:38:27.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:38:27.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:38:27.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:38:27.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:38:27.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:38:27.383 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:38:27.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:38:27.650 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:38:27.650 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:38:27.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:38:27.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:27.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:38:27.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836489 2021-07-26T06:38:27.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836489 2021-07-26T06:38:27.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489' 2021-07-26T06:38:27.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:27.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:38:27.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=176093659138 2021-07-26T06:38:27.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 176093659138 2021-07-26T06:38:27.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-176093659138' 2021-07-26T06:38:27.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:27.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:38:28.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=68719476743 2021-07-26T06:38:28.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 68719476743 2021-07-26T06:38:28.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-176093659138 2-68719476743' 2021-07-26T06:38:28.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:28.043 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836489 2021-07-26T06:38:28.043 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:28.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:38:28.045 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836489 2021-07-26T06:38:28.045 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:28.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836489 2021-07-26T06:38:28.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2021-07-26T06:38:28.046 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836489 2021-07-26T06:38:28.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:38:28.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836488 -lt 21474836489 2021-07-26T06:38:28.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:38:29.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:38:29.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:38:29.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836489 2021-07-26T06:38:29.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:29.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-176093659138 2021-07-26T06:38:29.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:29.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:38:29.595 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-176093659138 2021-07-26T06:38:29.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:29.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=176093659138 2021-07-26T06:38:29.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2021-07-26T06:38:29.597 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 176093659138 2021-07-26T06:38:29.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:38:29.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 176093659138 -lt 176093659138 2021-07-26T06:38:29.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:29.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-68719476743 2021-07-26T06:38:29.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:29.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:38:29.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-68719476743 2021-07-26T06:38:29.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:29.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=68719476743 2021-07-26T06:38:29.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 68719476743' 2021-07-26T06:38:29.914 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 68719476743 2021-07-26T06:38:29.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:38:30.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 68719476743 -lt 68719476743 2021-07-26T06:38:30.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:38:30.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:38:30.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:38:30.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:38:30.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:38:30.514 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:38:30.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:38:30.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:38:30.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:38:30.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:38:30.516 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:38:30.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:38:30.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:38:30.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:38:30.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:38:31.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:38:31.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:38:31.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:38:31.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:301: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2021-07-26T06:38:31.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:38:31.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:38:31.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:38:31.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:38:31.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:301: auto_repair_erasure_coded: local pgid=2.0 2021-07-26T06:38:31.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2021-07-26T06:38:31.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:38:31.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:38:31.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:38:31.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:38:31.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: auto_repair_erasure_coded: wait_for_scrub 2.0 2021-07-26T06:38:18.752744+0000 2021-07-26T06:38:31.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:38:31.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:38:18.752744+0000 2021-07-26T06:38:31.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:38:31.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:38:31.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:38:31.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:38:31.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:38:31.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:38:31.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:38:31.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:38:32.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:38:18.752744+0000 '>' 2021-07-26T06:38:18.752744+0000 2021-07-26T06:38:32.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:38:33.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:38:33.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:38:33.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:38:33.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:38:33.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:38:33.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:38:33.161 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:38:33.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:38:18.752744+0000 '>' 2021-07-26T06:38:18.752744+0000 2021-07-26T06:38:33.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:38:34.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:38:34.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:38:34.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:38:34.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:38:34.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:38:34.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:38:34.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:38:34.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:38:18.752744+0000 '>' 2021-07-26T06:38:18.752744+0000 2021-07-26T06:38:34.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:38:35.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:38:35.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:38:35.742 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:38:35.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:38:35.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:38:35.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:38:35.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:38:36.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:38:18.752744+0000 '>' 2021-07-26T06:38:18.752744+0000 2021-07-26T06:38:36.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:38:37.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:38:37.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:38:37.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:38:37.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:38:37.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:38:37.159 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:38:37.159 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:38:37.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:38:31.749554+0000 '>' 2021-07-26T06:38:18.752744+0000 2021-07-26T06:38:37.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:38:37.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: wait_for_clean 2021-07-26T06:38:37.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:38:37.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:38:37.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:38:37.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:38:37.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:38:37.435 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:38:37.435 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:38:37.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:38:37.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:38:37.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:38:37.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:38:37.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:38:37.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:38:37.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:38:37.526 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:38:37.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:38:37.791 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:38:37.791 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:38:37.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:38:37.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:37.792 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:38:37.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836492 2021-07-26T06:38:37.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836492 2021-07-26T06:38:37.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836492' 2021-07-26T06:38:37.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:37.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:38:38.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=176093659141 2021-07-26T06:38:38.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 176093659141 2021-07-26T06:38:38.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836492 1-176093659141' 2021-07-26T06:38:38.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:38.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:38:38.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=68719476746 2021-07-26T06:38:38.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 68719476746 2021-07-26T06:38:38.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836492 1-176093659141 2-68719476746' 2021-07-26T06:38:38.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:38.144 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836492 2021-07-26T06:38:38.144 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:38.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:38:38.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836492 2021-07-26T06:38:38.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:38.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836492 2021-07-26T06:38:38.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2021-07-26T06:38:38.147 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836492 2021-07-26T06:38:38.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:38:38.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836491 -lt 21474836492 2021-07-26T06:38:38.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:38:39.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:38:39.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:38:39.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836492 -lt 21474836492 2021-07-26T06:38:39.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:39.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-176093659141 2021-07-26T06:38:39.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:39.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:38:39.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-176093659141 2021-07-26T06:38:39.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:39.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=176093659141 2021-07-26T06:38:39.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 176093659141' 2021-07-26T06:38:39.806 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 176093659141 2021-07-26T06:38:39.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:38:40.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 176093659141 -lt 176093659141 2021-07-26T06:38:40.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:40.074 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-68719476746 2021-07-26T06:38:40.074 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:40.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:38:40.075 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-68719476746 2021-07-26T06:38:40.075 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:40.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=68719476746 2021-07-26T06:38:40.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 68719476746' 2021-07-26T06:38:40.077 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 68719476746 2021-07-26T06:38:40.077 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:38:40.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 68719476746 -lt 68719476746 2021-07-26T06:38:40.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:38:40.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:38:40.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:38:40.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:38:40.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:38:40.765 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:38:40.765 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:38:40.765 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:38:40.765 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:38:40.766 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:38:40.766 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:38:41.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:38:41.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:38:41.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:38:41.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:38:41.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:38:41.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:38:41.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:38:41.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:306: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2021-07-26T06:38:41.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=ecpool 2021-07-26T06:38:41.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:38:41.353 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary ecpool SOMETHING 2021-07-26T06:38:41.353 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=ecpool 2021-07-26T06:38:41.353 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:38:41.354 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:38:41.354 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:38:41.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=2 2021-07-26T06:38:41.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:38:41.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2021-07-26T06:38:42.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:306: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:38:42.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:38:42.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:38:42.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:38:42.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:38:42.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:38:42.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:38:42.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:38:42.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:38:42.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:38:42.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:38:42.067 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:38:42.067 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:38:42.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:38:42.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:38:42.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:38:42.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:38:42.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:38:42.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:38:42.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:38:42.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:38:42.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:38:42.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:38:42.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2021-07-26T06:38:43.096 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2021-07-26T06:38:43.097 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:38:43.097 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:38:43.098 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:38:43.098 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:38:43.098 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2021-07-26T06:38:43.098 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:38:43.099 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:38:43.099 INFO:tasks.workunit.client.0.smithi071.stdout:hinfo_key 2021-07-26T06:38:43.099 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:38:43.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:38:43.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:38:43.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:38:43.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:38:43.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:38:43.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:38:43.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:38:43.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:38:43.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:38:43.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:38:43.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:38:43.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:38:43.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:38:43.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:38:43.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:38:43.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:38:43.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:38:43.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:38:43.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:38:43.577 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:38:43.578 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:43.578 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:38:43.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:38:43.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:38:43.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:38:43.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:38:43.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:38:43.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:38:43.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:38:43.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:38:43.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:38:43.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:38:43.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:38:43.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:38:43.582 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:38:43.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:38:43.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:38:43.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:38:43.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:38:43.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:38:43.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:38:43.600 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:43.601+0000 7f78332710c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:43.619 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:43.620+0000 7f78332710c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:43.620 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:43.621+0000 7f78332710c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:43.621 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:43.622+0000 7f78332710c0 -1 Falling back to public interface 2021-07-26T06:38:43.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:38:43.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:38:43.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:38:43.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:38:43.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:38:43.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:43.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:38:43.856 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:38:43.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:43.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:38:44.096 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:44.097+0000 7f78332710c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:38:44.097 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:44.097+0000 7f78332710c0 -1 journal do_read_entry(737280): bad header magic 2021-07-26T06:38:44.097 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:44.097+0000 7f78332710c0 -1 journal do_read_entry(737280): bad header magic 2021-07-26T06:38:44.106 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:44.108+0000 7f78332710c0 -1 osd.1 51 log_to_monitors {default=true} 2021-07-26T06:38:44.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:38:45.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:38:45.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:45.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:38:45.138 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:38:45.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:45.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:38:45.535 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 52 last_clean_interval [41,51) [v2:127.0.0.1:6810/192564,v1:127.0.0.1:6811/192564] [v2:127.0.0.1:6812/192564,v1:127.0.0.1:6813/192564] exists,up 73c31ab6-d046-4604-9d36-67e7f791f953 2021-07-26T06:38:45.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:38:45.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:38:45.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:38:45.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:38:45.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:38:45.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:38:45.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:38:45.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:38:45.537 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:38:45.537 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:38:45.537 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:38:45.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:38:45.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:38:45.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:38:45.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:38:45.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:38:45.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:38:45.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:38:45.630 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:38:45.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:38:45.972 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:38:45.973 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:38:45.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:38:45.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:45.973 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:38:46.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836494 2021-07-26T06:38:46.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836494 2021-07-26T06:38:46.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494' 2021-07-26T06:38:46.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:46.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:38:46.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=236223201282 2021-07-26T06:38:46.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 236223201282 2021-07-26T06:38:46.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-236223201282' 2021-07-26T06:38:46.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:38:46.223 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:38:46.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=68719476749 2021-07-26T06:38:46.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 68719476749 2021-07-26T06:38:46.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-236223201282 2-68719476749' 2021-07-26T06:38:46.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:46.339 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836494 2021-07-26T06:38:46.339 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:46.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:38:46.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836494 2021-07-26T06:38:46.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:46.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836494 2021-07-26T06:38:46.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2021-07-26T06:38:46.342 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836494 2021-07-26T06:38:46.342 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:38:46.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836493 -lt 21474836494 2021-07-26T06:38:46.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:38:47.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:38:47.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:38:47.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836495 -lt 21474836494 2021-07-26T06:38:47.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:47.893 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-236223201282 2021-07-26T06:38:47.893 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:47.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:38:47.894 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-236223201282 2021-07-26T06:38:47.894 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:47.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=236223201282 2021-07-26T06:38:47.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 236223201282' 2021-07-26T06:38:47.896 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 236223201282 2021-07-26T06:38:47.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:38:48.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 236223201282 -lt 236223201282 2021-07-26T06:38:48.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:38:48.163 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-68719476749 2021-07-26T06:38:48.163 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:38:48.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:38:48.164 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-68719476749 2021-07-26T06:38:48.164 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:38:48.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=68719476749 2021-07-26T06:38:48.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 68719476749' 2021-07-26T06:38:48.166 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 68719476749 2021-07-26T06:38:48.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:38:48.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 68719476749 -lt 68719476749 2021-07-26T06:38:48.431 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:38:48.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:38:48.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:38:48.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:38:48.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:38:48.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:38:48.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:38:48.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:38:48.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:38:48.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:38:48.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:38:49.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:38:49.015 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:38:49.015 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:38:49.015 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:38:49.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:38:49.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:38:49.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:38:49.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:307: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:38:49.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:38:49.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:38:49.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:38:49.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:38:49.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:38:49.481 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:38:49.481 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:38:49.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:38:49.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:38:49.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:38:49.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:38:49.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:38:49.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:38:49.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:38:49.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:38:49.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:38:49.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:38:49.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:38:49.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:38:49.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:38:49.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:38:49.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:38:49.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:38:49.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:38:49.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:38:49.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:49.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:38:49.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:38:49.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:38:49.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:38:49.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:38:49.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:38:49.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:38:49.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:38:49.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:38:49.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:38:49.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:38:49.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:38:49.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:38:49.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:38:49.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:38:49.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:38:49.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:38:49.613 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:38:49.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:38:49.615 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:38:49.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:38:49.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:38:49.616 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:38:49.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:38:49.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:38:49.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:38:49.616 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:38:49.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:38:49.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:38:49.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:38:49.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:38:49.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:49.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:38:49.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:38:49.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:38:49.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:38:49.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:38:49.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:38:49.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:49.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:38:49.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:38:49.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:38:49.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:38:49.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:38:49.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:38:49.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2021-07-26T06:38:49.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2021-07-26T06:38:49.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:317: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2021-07-26T06:38:49.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:267: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2021-07-26T06:38:49.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local allow_overwrites=true 2021-07-26T06:38:49.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local poolname=ecpool 2021-07-26T06:38:49.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2021-07-26T06:38:49.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:38:49.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:38:49.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:38:49.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:38:49.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:38:49.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T06:38:50.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:38:50.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:38:50.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:38:50.027 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:38:50.028 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:50.028 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:38:50.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:38:50.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:38:50.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:38:50.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:38:50.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:38:50.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:38:50.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:38:50.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:38:50.332 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:38:50.332 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:38:50.332 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:38:50.332 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:38:50.332 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:50.333 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:38:50.333 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:38:50.333 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:38:50.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:38:50.433 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:38:50.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:38:50.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:38:50.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:38:50.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:38:50.435 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:38:50.435 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:38:50.435 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:38:50.436 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:38:50.436 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:50.436 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:38:50.436 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:38:50.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:38:50.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:38:50.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2021-07-26T06:38:50.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:38:50.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:38:50.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:38:50.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:38:50.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:38:50.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:38:51.424 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:38:51.424 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:38:51.425 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:38:51.425 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:38:51.425 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:51.426 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:38:51.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:38:51.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:38:51.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:38:51.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:278: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:38:51.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: seq 0 2 2021-07-26T06:38:51.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: for id in $(seq 0 2) 2021-07-26T06:38:51.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: '[' true = true ']' 2021-07-26T06:38:51.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:38:51.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:38:51.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:38:51.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:38:51.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:38:51.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:38:51.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:38:51.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:38:51.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:38:51.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:38:51.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:38:51.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:38:51.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:38:51.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:38:51.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:38:51.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:38:51.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:38:51.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:38:51.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:38:51.457 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:38:51.457 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:51.458 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:38:51.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:38:51.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:38:51.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:38:51.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:38:51.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:38:51.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:38:51.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:38:51.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:38:51.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:38:51.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:38:51.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:38:51.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:38:51.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:38:51.461 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:38:51.461 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 8dc724f0-06ef-44c7-bb87-8251385047b7 2021-07-26T06:38:51.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8dc724f0-06ef-44c7-bb87-8251385047b7 2021-07-26T06:38:51.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8dc724f0-06ef-44c7-bb87-8251385047b7' 2021-07-26T06:38:51.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:38:51.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB7WP5guxPvGxAA/P0V0/24wmmCwmyGrhO6aw== 2021-07-26T06:38:51.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB7WP5guxPvGxAA/P0V0/24wmmCwmyGrhO6aw=="}' 2021-07-26T06:38:51.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8dc724f0-06ef-44c7-bb87-8251385047b7 -i td/osd-scrub-repair/0/new.json 2021-07-26T06:38:51.721 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:38:51.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:38:51.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQB7WP5guxPvGxAA/P0V0/24wmmCwmyGrhO6aw== --osd-uuid 8dc724f0-06ef-44c7-bb87-8251385047b7 2021-07-26T06:38:51.760 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:51.760+0000 7f68ce6fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:51.761 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:51.761+0000 7f68ce6fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:51.762 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:51.762+0000 7f68ce6fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:51.763 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:51.763+0000 7f68ce6fc0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:38:51.763 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:51.763+0000 7f68ce6fc0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:38:51.764 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:51.763+0000 7f68ce6fc0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:38:51.862 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:51.862+0000 7f68ce6fc0c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T06:38:57.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:38:57.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:38:57.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:38:57.713 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:38:57.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:38:58.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:38:58.073 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:38:58.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:38:58.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:38:58.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:38:58.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:38:58.097 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:58.098+0000 7f0a1d4d60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:58.099 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:58.099+0000 7f0a1d4d60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:58.100 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:58.100+0000 7f0a1d4d60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:58.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:38:58.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:38:58.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:38:58.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:38:58.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:38:58.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:58.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:38:58.386 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:38:58.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:58.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:38:58.618 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:38:58.618+0000 7f0a1d4d60c0 -1 Falling back to public interface 2021-07-26T06:38:58.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:38:59.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:38:59.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:59.653 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:38:59.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:38:59.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:59.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:38:59.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:00.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:00.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:00.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:39:00.929 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:39:00.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:00.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:39:01.041 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:01.041+0000 7f0a1d4d60c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:39:01.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:02.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:02.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:02.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:39:02.223 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:39:02.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:02.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:39:02.527 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/194265,v1:127.0.0.1:6803/194265] [v2:127.0.0.1:6804/194265,v1:127.0.0.1:6805/194265] exists,up 8dc724f0-06ef-44c7-bb87-8251385047b7 2021-07-26T06:39:02.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:39:02.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:39:02.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:39:02.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: for id in $(seq 0 2) 2021-07-26T06:39:02.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: '[' true = true ']' 2021-07-26T06:39:02.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:39:02.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:39:02.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:39:02.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:39:02.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:39:02.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:39:02.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:39:02.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:39:02.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:39:02.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:39:02.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:39:02.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:39:02.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:39:02.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:39:02.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:39:02.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:39:02.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:39:02.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:39:02.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:39:02.538 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:39:02.539 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:02.539 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:39:02.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:39:02.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:39:02.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:39:02.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:39:02.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:39:02.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:39:02.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:39:02.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:39:02.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:39:02.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:39:02.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:39:02.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:39:02.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:39:02.543 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:39:02.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=19067336-e0af-45dd-9f3b-4ccd7b9d2672 2021-07-26T06:39:02.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 19067336-e0af-45dd-9f3b-4ccd7b9d2672' 2021-07-26T06:39:02.544 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 19067336-e0af-45dd-9f3b-4ccd7b9d2672 2021-07-26T06:39:02.544 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:39:02.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCGWP5gy+WUIBAAbFPjufhb/jxkEbSarYL/iw== 2021-07-26T06:39:02.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCGWP5gy+WUIBAAbFPjufhb/jxkEbSarYL/iw=="}' 2021-07-26T06:39:02.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 19067336-e0af-45dd-9f3b-4ccd7b9d2672 -i td/osd-scrub-repair/1/new.json 2021-07-26T06:39:02.918 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:39:02.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:39:02.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCGWP5gy+WUIBAAbFPjufhb/jxkEbSarYL/iw== --osd-uuid 19067336-e0af-45dd-9f3b-4ccd7b9d2672 2021-07-26T06:39:02.958 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:02.958+0000 7fc825a6d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:02.959 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:02.959+0000 7fc825a6d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:02.960 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:02.960+0000 7fc825a6d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:02.961 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:02.960+0000 7fc825a6d0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:39:02.961 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:02.960+0000 7fc825a6d0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:39:02.961 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:02.960+0000 7fc825a6d0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:39:03.020 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:03.021+0000 7fc825a6d0c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T06:39:09.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:39:09.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:39:09.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:39:09.014 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:39:09.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:39:09.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:39:09.356 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:39:09.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:39:09.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:39:09.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:39:09.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:39:09.381 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:09.382+0000 7f8deafd70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:09.382 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:09.384+0000 7f8deafd70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:09.383 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:09.385+0000 7f8deafd70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:39:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:39:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:39:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:39:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:39:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:09.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:39:09.625 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:39:09.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:09.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:39:09.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:09.900 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:09.901+0000 7f8deafd70c0 -1 Falling back to public interface 2021-07-26T06:39:10.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:10.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:10.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:39:10.892 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:39:10.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:10.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:39:11.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:12.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:12.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:12.169 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:39:12.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:39:12.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:12.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:39:12.376 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:12.377+0000 7f8deafd70c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:39:12.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:13.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:13.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:13.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:39:13.477 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:39:13.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:13.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:39:13.745 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/195664,v1:127.0.0.1:6811/195664] [v2:127.0.0.1:6812/195664,v1:127.0.0.1:6813/195664] exists,up 19067336-e0af-45dd-9f3b-4ccd7b9d2672 2021-07-26T06:39:13.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:39:13.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:39:13.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:39:13.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: for id in $(seq 0 2) 2021-07-26T06:39:13.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: '[' true = true ']' 2021-07-26T06:39:13.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:39:13.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:39:13.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:39:13.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:39:13.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:39:13.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:39:13.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:39:13.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:39:13.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:39:13.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:39:13.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:39:13.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:39:13.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:39:13.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:39:13.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:39:13.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:39:13.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:39:13.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:39:13.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:39:13.755 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:39:13.756 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:13.756 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:39:13.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:39:13.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:39:13.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:39:13.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:39:13.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:39:13.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:39:13.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:39:13.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:39:13.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:39:13.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:39:13.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:39:13.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:39:13.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:39:13.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:39:13.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5db7cfcd-d998-4fe5-b6a6-28869a723fdf 2021-07-26T06:39:13.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5db7cfcd-d998-4fe5-b6a6-28869a723fdf' 2021-07-26T06:39:13.760 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 5db7cfcd-d998-4fe5-b6a6-28869a723fdf 2021-07-26T06:39:13.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:39:13.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRWP5gTBuLLRAAfpydkEE+sM/sw8KgRKX3Aw== 2021-07-26T06:39:13.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRWP5gTBuLLRAAfpydkEE+sM/sw8KgRKX3Aw=="}' 2021-07-26T06:39:13.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5db7cfcd-d998-4fe5-b6a6-28869a723fdf -i td/osd-scrub-repair/2/new.json 2021-07-26T06:39:14.179 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:39:14.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2021-07-26T06:39:14.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCRWP5gTBuLLRAAfpydkEE+sM/sw8KgRKX3Aw== --osd-uuid 5db7cfcd-d998-4fe5-b6a6-28869a723fdf 2021-07-26T06:39:14.220 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:14.221+0000 7f278a0270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:14.221 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:14.222+0000 7f278a0270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:14.222 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:14.223+0000 7f278a0270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:14.223 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:14.223+0000 7f278a0270c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:39:14.223 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:14.224+0000 7f278a0270c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:39:14.223 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:14.224+0000 7f278a0270c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:39:14.327 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:14.328+0000 7f278a0270c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2021-07-26T06:39:21.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T06:39:21.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:39:21.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:39:21.092 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:39:21.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:39:21.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:39:21.431 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:39:21.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:39:21.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:39:21.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:39:21.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:39:21.456 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:21.457+0000 7f86a81960c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:21.458 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:21.459+0000 7f86a81960c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:21.459 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:21.460+0000 7f86a81960c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:21.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:39:21.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:39:21.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:39:21.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:39:21.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:39:21.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:21.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:39:21.745 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:39:21.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:21.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:39:21.973 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:21.975+0000 7f86a81960c0 -1 Falling back to public interface 2021-07-26T06:39:22.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:23.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:23.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:23.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:39:23.029 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:39:23.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:23.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:39:23.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:24.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:24.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:24.306 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:39:24.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:39:24.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:24.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:39:24.494 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:24.495+0000 7f86a81960c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:39:24.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:25.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:25.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:25.651 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:39:25.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:39:25.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:25.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:39:25.997 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/197071,v1:127.0.0.1:6819/197071] [v2:127.0.0.1:6820/197071,v1:127.0.0.1:6821/197071] exists,up 5db7cfcd-d998-4fe5-b6a6-28869a723fdf 2021-07-26T06:39:25.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:39:25.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:39:25.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:39:26.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: auto_repair_erasure_coded: create_rbd_pool 2021-07-26T06:39:26.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:39:26.289 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T06:39:26.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:39:26.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:39:26.841 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T06:39:26.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:39:27.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:39:30.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: wait_for_clean 2021-07-26T06:39:30.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:39:30.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:39:30.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:39:30.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:39:30.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:39:30.054 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:39:30.054 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:39:30.055 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:39:30.055 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:39:30.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:39:30.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:39:30.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:39:30.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:39:30.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:39:30.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:39:30.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:39:30.428 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:39:30.428 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:39:30.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:39:30.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:39:30.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:39:30.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836487 2021-07-26T06:39:30.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836487 2021-07-26T06:39:30.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487' 2021-07-26T06:39:30.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:39:30.548 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:39:30.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=47244640261 2021-07-26T06:39:30.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 47244640261 2021-07-26T06:39:30.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-47244640261' 2021-07-26T06:39:30.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:39:30.720 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:39:30.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=90194313218 2021-07-26T06:39:30.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 90194313218 2021-07-26T06:39:30.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-47244640261 2-90194313218' 2021-07-26T06:39:30.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:39:30.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836487 2021-07-26T06:39:30.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:39:30.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:39:30.841 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836487 2021-07-26T06:39:30.841 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:39:30.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836487 2021-07-26T06:39:30.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-07-26T06:39:30.842 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836487 2021-07-26T06:39:30.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:39:31.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836487 2021-07-26T06:39:31.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:39:32.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:39:32.161 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:39:32.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836487 2021-07-26T06:39:32.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:39:32.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-47244640261 2021-07-26T06:39:32.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:39:32.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:39:32.440 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-47244640261 2021-07-26T06:39:32.440 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:39:32.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=47244640261 2021-07-26T06:39:32.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 47244640261' 2021-07-26T06:39:32.441 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 47244640261 2021-07-26T06:39:32.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:39:32.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 47244640261 -lt 47244640261 2021-07-26T06:39:32.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:39:32.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-90194313218 2021-07-26T06:39:32.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:39:32.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:39:32.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-90194313218 2021-07-26T06:39:32.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:39:32.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=90194313218 2021-07-26T06:39:32.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 90194313218' 2021-07-26T06:39:32.731 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 90194313218 2021-07-26T06:39:32.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:39:33.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 90194313219 -lt 90194313218 2021-07-26T06:39:33.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:39:33.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:39:33.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:39:33.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:39:33.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:39:33.370 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:39:33.370 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:39:33.370 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:39:33.370 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:39:33.371 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:39:33.371 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:39:33.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:39:33.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:39:33.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:39:33.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:39:33.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:39:33.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:39:33.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:39:33.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2021-07-26T06:39:33.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=ecpool 2021-07-26T06:39:33.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T06:39:33.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=true 2021-07-26T06:39:33.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T06:39:33.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2021-07-26T06:39:34.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2021-07-26T06:39:34.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2021-07-26T06:39:35.167 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' created 2021-07-26T06:39:35.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:39:36.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' true = true ']' 2021-07-26T06:39:36.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2021-07-26T06:39:36.959 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 2 allow_ec_overwrites to true 2021-07-26T06:39:36.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T06:39:36.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:39:36.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:39:36.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:39:36.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:39:36.973 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:39:36.973 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:39:36.973 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:39:36.973 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:39:36.973 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:39:36.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:39:37.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:39:37.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:39:37.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:39:37.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:39:37.070 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:39:37.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:39:37.487 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:39:37.488 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:39:37.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:39:37.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:39:37.488 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:39:37.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836490 2021-07-26T06:39:37.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836490 2021-07-26T06:39:37.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836490' 2021-07-26T06:39:37.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:39:37.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:39:37.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=47244640263 2021-07-26T06:39:37.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 47244640263 2021-07-26T06:39:37.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836490 1-47244640263' 2021-07-26T06:39:37.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:39:37.748 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:39:37.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=90194313221 2021-07-26T06:39:37.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 90194313221 2021-07-26T06:39:37.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836490 1-47244640263 2-90194313221' 2021-07-26T06:39:37.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:39:37.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836490 2021-07-26T06:39:37.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:39:37.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:39:37.868 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836490 2021-07-26T06:39:37.868 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:39:37.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836490 2021-07-26T06:39:37.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-07-26T06:39:37.869 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836490 2021-07-26T06:39:37.869 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:39:38.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836488 -lt 21474836490 2021-07-26T06:39:38.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:39:39.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:39:39.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:39:39.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836490 -lt 21474836490 2021-07-26T06:39:39.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:39:39.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-47244640263 2021-07-26T06:39:39.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:39:39.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:39:39.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-47244640263 2021-07-26T06:39:39.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:39:39.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=47244640263 2021-07-26T06:39:39.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 47244640263' 2021-07-26T06:39:39.416 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 47244640263 2021-07-26T06:39:39.416 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:39:39.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 47244640263 -lt 47244640263 2021-07-26T06:39:39.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:39:39.682 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-90194313221 2021-07-26T06:39:39.682 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:39:39.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:39:39.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-90194313221 2021-07-26T06:39:39.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:39:39.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=90194313221 2021-07-26T06:39:39.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 90194313221' 2021-07-26T06:39:39.685 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 90194313221 2021-07-26T06:39:39.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:39:39.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 90194313221 -lt 90194313221 2021-07-26T06:39:39.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:39:39.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:39:39.985 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:39:40.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:39:40.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:39:40.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:39:40.321 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:39:40.321 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:39:40.321 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:39:40.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:39:40.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:39:40.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:39:40.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:39:40.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:39:40.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:39:40.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:39:40.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:39:40.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:39:40.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T06:39:40.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: auto_repair_erasure_coded: local payload=ABCDEF 2021-07-26T06:39:40.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:294: auto_repair_erasure_coded: echo ABCDEF 2021-07-26T06:39:40.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:295: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T06:39:41.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2021-07-26T06:39:41.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=ecpool 2021-07-26T06:39:41.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:39:41.057 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary ecpool SOMETHING 2021-07-26T06:39:41.057 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=ecpool 2021-07-26T06:39:41.057 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:39:41.057 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:39:41.058 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:39:41.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=2 2021-07-26T06:39:41.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:39:41.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2021-07-26T06:39:41.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:39:41.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:39:41.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:39:41.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:39:41.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:39:41.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:39:41.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:39:41.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:39:41.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:39:41.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:39:41.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:39:41.689 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:39:41.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:39:41.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:39:41.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:39:41.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:39:41.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:39:41.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:39:41.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:39:41.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:39:41.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:39:41.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:39:41.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:39:41.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2021-07-26T06:39:44.295 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2021-07-26T06:39:44.295 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:39:44.296 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:39:44.296 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:39:44.296 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2021-07-26T06:39:44.296 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:39:44.296 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:39:44.297 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:39:44.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:39:44.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:39:44.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:39:44.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:39:44.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:39:44.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:39:44.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:39:44.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:39:44.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:39:44.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:39:44.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:39:44.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:39:44.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:39:44.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:39:44.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:39:44.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:39:44.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:39:44.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:39:44.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:39:44.820 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:39:44.820 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:44.820 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:39:44.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:39:44.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:39:44.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:39:44.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:39:44.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:39:44.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:39:44.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:39:44.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:39:44.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:39:44.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:39:44.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:39:44.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:39:44.823 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:39:44.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:39:44.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:39:44.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:39:44.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:39:44.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:39:44.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:39:44.843 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:44.843+0000 7f0eb45400c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:44.847 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:44.845+0000 7f0eb45400c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:44.847 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:44.846+0000 7f0eb45400c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:45.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:39:45.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:39:45.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:39:45.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:39:45.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:39:45.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:45.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:39:45.097 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:39:45.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:45.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:39:45.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:45.371 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:45.372+0000 7f0eb45400c0 -1 Falling back to public interface 2021-07-26T06:39:46.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:46.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:46.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:39:46.366 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:39:46.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:46.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:39:46.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:47.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:47.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:47.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:39:47.647 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:39:47.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:47.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:39:47.792 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:39:47.792+0000 7f0eb45400c0 -1 osd.1 41 log_to_monitors {default=true} 2021-07-26T06:39:47.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:48.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:48.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:48.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:39:48.922 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:39:48.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:48.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:39:49.345 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 42 last_clean_interval [11,41) [v2:127.0.0.1:6810/199008,v1:127.0.0.1:6811/199008] [v2:127.0.0.1:6812/199008,v1:127.0.0.1:6813/199008] exists,up 19067336-e0af-45dd-9f3b-4ccd7b9d2672 2021-07-26T06:39:49.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:39:49.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:39:49.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:39:49.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:39:49.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:39:49.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:39:49.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:39:49.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:39:49.348 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:39:49.348 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:39:49.348 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:39:49.348 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:39:49.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:39:49.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:39:49.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:39:49.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:39:49.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:39:49.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:39:49.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:39:49.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:39:49.796 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:39:49.796 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:39:49.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:39:49.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:39:49.797 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:39:50.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836493 2021-07-26T06:39:50.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836493 2021-07-26T06:39:50.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836493' 2021-07-26T06:39:50.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:39:50.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:39:50.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=206158430210 2021-07-26T06:39:50.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 206158430210 2021-07-26T06:39:50.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836493 1-206158430210' 2021-07-26T06:39:50.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:39:50.216 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:39:50.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=90194313224 2021-07-26T06:39:50.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 90194313224 2021-07-26T06:39:50.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836493 1-206158430210 2-90194313224' 2021-07-26T06:39:50.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:39:50.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836493 2021-07-26T06:39:50.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:39:50.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:39:50.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836493 2021-07-26T06:39:50.337 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:39:50.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836493 2021-07-26T06:39:50.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2021-07-26T06:39:50.338 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836493 2021-07-26T06:39:50.338 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:39:50.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836493 -lt 21474836493 2021-07-26T06:39:50.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:39:50.644 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-206158430210 2021-07-26T06:39:50.645 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:39:50.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:39:50.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-206158430210 2021-07-26T06:39:50.647 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:39:50.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=206158430210 2021-07-26T06:39:50.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 206158430210' 2021-07-26T06:39:50.648 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 206158430210 2021-07-26T06:39:50.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:39:50.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 206158430210 2021-07-26T06:39:50.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:39:51.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:39:51.953 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:39:52.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 206158430210 -lt 206158430210 2021-07-26T06:39:52.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:39:52.232 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-90194313224 2021-07-26T06:39:52.232 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:39:52.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:39:52.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-90194313224 2021-07-26T06:39:52.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:39:52.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=90194313224 2021-07-26T06:39:52.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 90194313224' 2021-07-26T06:39:52.236 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 90194313224 2021-07-26T06:39:52.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:39:52.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 90194313225 -lt 90194313224 2021-07-26T06:39:52.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:39:52.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:39:52.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:39:52.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:39:52.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:39:52.856 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:39:52.856 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:39:52.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:39:52.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:39:52.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:39:52.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:39:53.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:39:53.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:39:53.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:39:53.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:39:53.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:39:53.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:39:53.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:39:53.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:301: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2021-07-26T06:39:53.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:39:53.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:39:53.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:39:53.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:39:53.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:301: auto_repair_erasure_coded: local pgid=2.0 2021-07-26T06:39:53.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2021-07-26T06:39:53.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:39:53.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:39:53.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:39:53.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:39:54.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: auto_repair_erasure_coded: wait_for_scrub 2.0 2021-07-26T06:39:35.030916+0000 2021-07-26T06:39:54.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:39:54.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:39:35.030916+0000 2021-07-26T06:39:54.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:39:54.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:39:54.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:39:54.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:39:54.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:39:54.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:39:54.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:39:54.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:39:54.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:39:35.030916+0000 '>' 2021-07-26T06:39:35.030916+0000 2021-07-26T06:39:54.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:39:55.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:39:55.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:39:55.385 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:39:55.385 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:39:55.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:39:55.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:39:55.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:39:55.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:39:35.030916+0000 '>' 2021-07-26T06:39:35.030916+0000 2021-07-26T06:39:55.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:39:56.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:39:56.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:39:56.748 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:39:56.748 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:39:56.749 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:39:56.749 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:39:56.749 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:39:56.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:39:35.030916+0000 '>' 2021-07-26T06:39:35.030916+0000 2021-07-26T06:39:56.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:39:57.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:39:57.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:39:57.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:39:57.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:39:57.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:39:57.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:39:57.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:39:58.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:39:53.439883+0000 '>' 2021-07-26T06:39:35.030916+0000 2021-07-26T06:39:58.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:39:58.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: wait_for_clean 2021-07-26T06:39:58.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:39:58.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:39:58.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:39:58.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:39:58.220 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:39:58.220 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:39:58.220 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:39:58.220 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:39:58.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:39:58.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:39:58.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:39:58.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:39:58.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:39:58.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:39:58.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:39:58.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:39:58.577 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:39:58.577 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:39:58.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:39:58.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:39:58.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:39:58.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836496 2021-07-26T06:39:58.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836496 2021-07-26T06:39:58.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496' 2021-07-26T06:39:58.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:39:58.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:39:58.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=206158430213 2021-07-26T06:39:58.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 206158430213 2021-07-26T06:39:58.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-206158430213' 2021-07-26T06:39:58.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:39:58.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:39:58.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=90194313227 2021-07-26T06:39:58.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 90194313227 2021-07-26T06:39:58.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-206158430213 2-90194313227' 2021-07-26T06:39:58.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:39:58.972 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836496 2021-07-26T06:39:58.972 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:39:58.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:39:58.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836496 2021-07-26T06:39:58.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:39:58.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836496 2021-07-26T06:39:58.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-07-26T06:39:58.975 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836496 2021-07-26T06:39:58.976 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:39:59.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836495 -lt 21474836496 2021-07-26T06:39:59.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:40:00.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:40:00.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:00.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836496 -lt 21474836496 2021-07-26T06:40:00.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:00.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-206158430213 2021-07-26T06:40:00.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:00.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:40:00.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-206158430213 2021-07-26T06:40:00.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:00.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=206158430213 2021-07-26T06:40:00.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 206158430213' 2021-07-26T06:40:00.543 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 206158430213 2021-07-26T06:40:00.544 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:40:00.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 206158430213 -lt 206158430213 2021-07-26T06:40:00.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:00.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-90194313227 2021-07-26T06:40:00.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:00.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:40:00.898 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-90194313227 2021-07-26T06:40:00.898 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:00.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=90194313227 2021-07-26T06:40:00.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 90194313227' 2021-07-26T06:40:00.899 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 90194313227 2021-07-26T06:40:00.899 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:40:01.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 90194313227 -lt 90194313227 2021-07-26T06:40:01.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:40:01.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:40:01.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:40:01.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:40:01.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:40:01.533 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:40:01.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:40:01.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:40:01.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:40:01.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:40:01.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:40:01.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:40:01.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:40:01.796 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:40:01.796 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:40:02.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:40:02.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:40:02.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:40:02.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:306: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2021-07-26T06:40:02.128 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=ecpool 2021-07-26T06:40:02.128 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:40:02.129 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary ecpool SOMETHING 2021-07-26T06:40:02.129 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=ecpool 2021-07-26T06:40:02.129 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:40:02.130 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:40:02.130 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:40:02.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=2 2021-07-26T06:40:02.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:40:02.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2021-07-26T06:40:02.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:306: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:40:02.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:40:02.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:40:02.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:40:02.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:40:02.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:40:02.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:40:02.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:40:02.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:40:02.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:40:02.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:40:02.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:40:02.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:40:02.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:40:02.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:40:02.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:40:02.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:40:02.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:40:02.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:40:02.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:40:02.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:40:02.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:40:02.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:40:02.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2021-07-26T06:40:04.676 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2021-07-26T06:40:04.677 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:40:04.677 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:40:04.677 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:40:04.677 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2021-07-26T06:40:04.678 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:40:04.678 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:40:04.678 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:40:04.678 INFO:tasks.workunit.client.0.smithi071.stdout:hinfo_key 2021-07-26T06:40:04.679 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:40:04.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:40:04.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:40:04.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:40:04.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:40:04.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:40:04.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:40:04.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:40:04.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:40:04.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:40:04.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:40:04.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:40:04.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:40:04.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:40:04.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:40:04.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:40:04.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:40:04.953 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:40:04.953 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:40:04.953 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:40:04.954 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:40:04.954 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:40:04.954 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:40:04.954 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:04.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:04.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:40:04.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:40:04.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:40:04.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:40:04.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:40:04.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:40:04.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:40:04.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T06:40:04.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:40:04.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:40:04.957 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:40:04.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2021-07-26T06:40:04.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:40:04.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:40:04.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:40:04.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:40:04.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:40:04.976 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:04.976+0000 7feb98d8e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:04.978 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:04.978+0000 7feb98d8e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:04.979 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:04.979+0000 7feb98d8e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:05.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:40:05.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:40:05.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:40:05.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:40:05.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:40:05.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:05.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:40:05.228 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:40:05.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:05.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:40:05.499 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:05.499+0000 7feb98d8e0c0 -1 Falling back to public interface 2021-07-26T06:40:05.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:40:06.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:40:06.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:06.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:40:06.507 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:40:06.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:06.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:40:06.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:40:07.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:40:07.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:07.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:40:07.843 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:40:07.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:07.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:40:07.941 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:07.942+0000 7feb98d8e0c0 -1 osd.1 57 log_to_monitors {default=true} 2021-07-26T06:40:08.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:40:09.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:40:09.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:09.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:40:09.189 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:40:09.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:09.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:40:09.529 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 63 up_thru 63 down_at 58 last_clean_interval [48,57) [v2:127.0.0.1:6810/200948,v1:127.0.0.1:6811/200948] [v2:127.0.0.1:6812/200948,v1:127.0.0.1:6813/200948] exists,up 19067336-e0af-45dd-9f3b-4ccd7b9d2672 2021-07-26T06:40:09.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:40:09.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:40:09.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:40:09.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:40:09.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:40:09.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:40:09.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:40:09.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:40:09.531 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:40:09.531 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:40:09.531 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:40:09.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:40:09.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:40:09.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:40:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:40:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:40:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:40:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:40:09.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:40:09.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:40:09.954 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:40:09.955 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:40:09.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:40:09.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:09.955 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:40:10.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836499 2021-07-26T06:40:10.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836499 2021-07-26T06:40:10.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836499' 2021-07-26T06:40:10.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:10.175 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:40:10.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=270582939650 2021-07-26T06:40:10.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 270582939650 2021-07-26T06:40:10.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836499 1-270582939650' 2021-07-26T06:40:10.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:10.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:40:10.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=90194313230 2021-07-26T06:40:10.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 90194313230 2021-07-26T06:40:10.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836499 1-270582939650 2-90194313230' 2021-07-26T06:40:10.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:10.416 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836499 2021-07-26T06:40:10.416 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:10.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:40:10.418 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836499 2021-07-26T06:40:10.418 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:10.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836499 2021-07-26T06:40:10.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2021-07-26T06:40:10.419 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836499 2021-07-26T06:40:10.419 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:10.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836498 -lt 21474836499 2021-07-26T06:40:10.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:40:11.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:40:11.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:11.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836498 -lt 21474836499 2021-07-26T06:40:11.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:40:12.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:40:12.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:13.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836499 -lt 21474836499 2021-07-26T06:40:13.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:13.231 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-270582939650 2021-07-26T06:40:13.231 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:13.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:40:13.232 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-270582939650 2021-07-26T06:40:13.232 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:13.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=270582939650 2021-07-26T06:40:13.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 270582939650' 2021-07-26T06:40:13.234 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 270582939650 2021-07-26T06:40:13.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:40:13.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 270582939650 -lt 270582939650 2021-07-26T06:40:13.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:13.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-90194313230 2021-07-26T06:40:13.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:13.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:40:13.533 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-90194313230 2021-07-26T06:40:13.533 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:13.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=90194313230 2021-07-26T06:40:13.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 90194313230' 2021-07-26T06:40:13.535 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 90194313230 2021-07-26T06:40:13.535 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:40:13.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 90194313231 -lt 90194313230 2021-07-26T06:40:13.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:40:13.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:40:13.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:40:14.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:40:14.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:40:14.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:40:14.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:40:14.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:40:14.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:40:14.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:40:14.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:40:14.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:40:14.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:40:14.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:40:14.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:40:14.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:40:14.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:40:14.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:40:14.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:307: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:40:14.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:40:14.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:40:14.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:40:14.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:40:14.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:40:14.850 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:40:14.851 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:40:14.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:40:14.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:40:14.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:40:14.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:40:14.954 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:40:14.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:40:14.956 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:40:14.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:40:14.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:40:14.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:40:14.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:40:14.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:40:14.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:40:14.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:40:14.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:40:14.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:40:14.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:40:14.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:40:14.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:40:14.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:40:14.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:40:14.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:40:14.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:40:14.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:40:14.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:40:14.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:40:14.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:40:14.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:40:14.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:40:14.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:40:14.979 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:40:14.979 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:40:14.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:40:14.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:40:14.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:40:14.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:40:14.980 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:40:14.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:40:14.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:40:14.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:40:14.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:40:14.986 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:40:14.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:40:14.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:40:14.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:40:14.987 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:40:14.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:40:14.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:40:14.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:40:14.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:40:14.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:40:14.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:40:14.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:40:14.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:40:14.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:40:14.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:40:14.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:40:14.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:40:14.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:40:14.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:40:14.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:40:14.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:40:14.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:40:14.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:40:14.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2021-07-26T06:40:14.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:816: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2021-07-26T06:40:14.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:795: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2021-07-26T06:40:14.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: corrupt_and_repair_jerasure: local allow_overwrites=false 2021-07-26T06:40:14.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:797: corrupt_and_repair_jerasure: local poolname=ecpool 2021-07-26T06:40:14.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2021-07-26T06:40:14.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:40:14.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:40:14.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:40:14.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:40:14.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:40:14.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T06:40:15.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:40:15.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:40:15.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:40:15.392 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:40:15.393 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:40:15.393 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:40:15.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:15.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:40:15.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:40:15.680 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:40:15.681 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:40:15.681 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:40:15.681 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:40:15.681 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:40:15.682 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:40:15.682 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:40:15.683 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:40:15.683 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:40:15.683 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:40:15.684 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:40:15.684 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:40:15.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:40:15.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:40:15.778 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:40:15.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:40:15.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:40:15.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:40:15.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:40:15.780 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:40:15.780 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:40:15.780 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:40:15.780 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:40:15.781 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:40:15.781 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:40:15.781 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:40:15.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:40:15.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:40:15.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2021-07-26T06:40:15.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:40:15.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:40:15.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:40:15.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:40:15.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:40:15.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:40:16.715 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:40:16.715 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:40:16.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:40:16.716 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:40:16.716 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:40:16.716 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:40:16.717 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:16.717 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:40:16.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:40:16.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: corrupt_and_repair_jerasure: seq 0 3 2021-07-26T06:40:16.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2021-07-26T06:40:16.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: '[' false = true ']' 2021-07-26T06:40:16.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd_filestore td/osd-scrub-repair 0 2021-07-26T06:40:16.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:40:16.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:40:16.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2021-07-26T06:40:16.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:40:16.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:40:16.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:40:16.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:40:16.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:40:16.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:40:16.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:40:16.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:40:16.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:40:16.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:40:16.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:40:16.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:40:16.750 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:40:16.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:40:16.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:40:16.751 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:40:16.752 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:40:16.752 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:40:16.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:16.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:16.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:40:16.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:40:16.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:40:16.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:40:16.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:40:16.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:40:16.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:40:16.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:40:16.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:40:16.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:40:16.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:40:16.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:40:16.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=00dcb6d2-aaef-4520-a931-7ecbbd361c32 2021-07-26T06:40:16.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd2 00dcb6d2-aaef-4520-a931-7ecbbd361c32' 2021-07-26T06:40:16.756 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 00dcb6d2-aaef-4520-a931-7ecbbd361c32 2021-07-26T06:40:16.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:40:16.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQDQWP5gRU1GLRAAQjTykSYaAIrfp0kPuvBZoQ== 2021-07-26T06:40:16.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQDQWP5gRU1GLRAAQjTykSYaAIrfp0kPuvBZoQ=="}' 2021-07-26T06:40:16.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 00dcb6d2-aaef-4520-a931-7ecbbd361c32 -i td/osd-scrub-repair/0/new.json 2021-07-26T06:40:17.037 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:40:17.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:40:17.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDQWP5gRU1GLRAAQjTykSYaAIrfp0kPuvBZoQ== --osd-uuid 00dcb6d2-aaef-4520-a931-7ecbbd361c32 --osd-objectstore=filestore 2021-07-26T06:40:17.073 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:17.074+0000 7f17c944a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:17.074 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:17.076+0000 7f17c944a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:17.075 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:17.077+0000 7f17c944a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:17.715 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:17.716+0000 7f17c944a0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:40:18.234 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:18.235+0000 7f17c944a0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:40:18.235 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:18.236+0000 7f17c944a0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:40:18.235 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:18.236+0000 7f17c944a0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:40:18.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:40:18.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:40:18.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd0 key to auth repository 2021-07-26T06:40:18.737 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:40:18.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:40:18.910 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2021-07-26T06:40:18.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.0 2021-07-26T06:40:18.912 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:40:18.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:40:18.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:40:18.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:40:18.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:40:18.934 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:18.935+0000 7ffbad1730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:18.936 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:18.937+0000 7ffbad1730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:18.937 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:18.938+0000 7ffbad1730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:18.938 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:18.939+0000 7ffbad1730c0 -1 Falling back to public interface 2021-07-26T06:40:19.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 0 2021-07-26T06:40:19.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:40:19.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:40:19.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:40:19.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:40:19.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:19.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:40:19.197 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:40:19.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:19.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:40:19.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:40:19.519 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:19.520+0000 7ffbad1730c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:40:19.532 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:19.533+0000 7ffbad1730c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:40:19.532 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:19.533+0000 7ffbad1730c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:40:19.537 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:19.539+0000 7ffbad1730c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:40:20.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:40:20.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:20.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:40:20.376 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:40:20.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:20.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:40:20.561 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/202289,v1:127.0.0.1:6801/202289] [v2:127.0.0.1:6802/202289,v1:127.0.0.1:6803/202289] exists,up 00dcb6d2-aaef-4520-a931-7ecbbd361c32 2021-07-26T06:40:20.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:40:20.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:40:20.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:40:20.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2021-07-26T06:40:20.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: '[' false = true ']' 2021-07-26T06:40:20.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd_filestore td/osd-scrub-repair 1 2021-07-26T06:40:20.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:40:20.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:40:20.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2021-07-26T06:40:20.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:40:20.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:40:20.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:40:20.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:40:20.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:40:20.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:40:20.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:40:20.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:40:20.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:40:20.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:40:20.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:40:20.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:40:20.572 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:40:20.572 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:40:20.572 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:40:20.572 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:40:20.573 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:40:20.573 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:40:20.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:20.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:20.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:40:20.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:40:20.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:40:20.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:40:20.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:40:20.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:40:20.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:40:20.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:40:20.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:40:20.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:40:20.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:40:20.576 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:40:20.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=31c57fb5-51ff-4cb0-a76a-87197652a01e 2021-07-26T06:40:20.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd2 31c57fb5-51ff-4cb0-a76a-87197652a01e' 2021-07-26T06:40:20.577 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 31c57fb5-51ff-4cb0-a76a-87197652a01e 2021-07-26T06:40:20.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:40:20.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQDUWP5gJ2GqIhAAWIjuBLVHlCeXpzphW5Ojdw== 2021-07-26T06:40:20.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQDUWP5gJ2GqIhAAWIjuBLVHlCeXpzphW5Ojdw=="}' 2021-07-26T06:40:20.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 31c57fb5-51ff-4cb0-a76a-87197652a01e -i td/osd-scrub-repair/1/new.json 2021-07-26T06:40:21.048 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:40:21.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:40:21.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDUWP5gJ2GqIhAAWIjuBLVHlCeXpzphW5Ojdw== --osd-uuid 31c57fb5-51ff-4cb0-a76a-87197652a01e --osd-objectstore=filestore 2021-07-26T06:40:21.085 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:21.086+0000 7fc43b1720c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:21.087 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:21.088+0000 7fc43b1720c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:21.088 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:21.089+0000 7fc43b1720c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:21.801 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:21.802+0000 7fc43b1720c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:40:22.665 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:22.666+0000 7fc43b1720c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:40:22.679 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:22.680+0000 7fc43b1720c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:40:22.679 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:22.680+0000 7fc43b1720c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:40:23.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:40:23.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:40:23.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd1 key to auth repository 2021-07-26T06:40:23.424 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:40:23.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:40:23.756 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2021-07-26T06:40:23.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.1 2021-07-26T06:40:23.758 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:40:23.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:40:23.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:40:23.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:40:23.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:40:23.779 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:23.780+0000 7fd2acef70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:23.781 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:23.782+0000 7fd2acef70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:23.782 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:23.783+0000 7fd2acef70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:23.783 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:23.784+0000 7fd2acef70c0 -1 Falling back to public interface 2021-07-26T06:40:24.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 1 2021-07-26T06:40:24.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:40:24.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:40:24.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:40:24.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:40:24.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:24.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:40:24.137 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:40:24.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:24.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:40:24.298 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:24.299+0000 7fd2acef70c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:40:24.302 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:24.303+0000 7fd2acef70c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:40:24.302 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:24.303+0000 7fd2acef70c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:40:24.309 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:24.311+0000 7fd2acef70c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:40:24.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:40:25.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:40:25.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:25.441 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:40:25.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:40:25.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:25.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:40:25.791 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/202658,v1:127.0.0.1:6811/202658] [v2:127.0.0.1:6812/202658,v1:127.0.0.1:6813/202658] exists,up 31c57fb5-51ff-4cb0-a76a-87197652a01e 2021-07-26T06:40:25.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:40:25.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:40:25.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:40:25.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2021-07-26T06:40:25.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: '[' false = true ']' 2021-07-26T06:40:25.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd_filestore td/osd-scrub-repair 2 2021-07-26T06:40:25.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:40:25.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:40:25.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2021-07-26T06:40:25.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:40:25.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:40:25.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:40:25.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:40:25.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:40:25.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:40:25.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:40:25.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:40:25.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:40:25.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:40:25.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:40:25.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:40:25.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:40:25.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:40:25.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:40:25.803 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:40:25.803 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:40:25.804 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:40:25.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:25.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:25.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:40:25.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:40:25.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:40:25.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:40:25.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:40:25.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:40:25.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:40:25.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:40:25.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:40:25.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:40:25.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:40:25.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:40:25.807 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 780522fb-cdae-4eb2-80e8-854a9ba99cbb 2021-07-26T06:40:25.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=780522fb-cdae-4eb2-80e8-854a9ba99cbb 2021-07-26T06:40:25.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd2 780522fb-cdae-4eb2-80e8-854a9ba99cbb' 2021-07-26T06:40:25.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:40:25.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQDZWP5gVnVVMBAAFQMXNiBz59XEpYAM62v6gQ== 2021-07-26T06:40:25.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQDZWP5gVnVVMBAAFQMXNiBz59XEpYAM62v6gQ=="}' 2021-07-26T06:40:25.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 780522fb-cdae-4eb2-80e8-854a9ba99cbb -i td/osd-scrub-repair/2/new.json 2021-07-26T06:40:26.157 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:40:26.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2021-07-26T06:40:26.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDZWP5gVnVVMBAAFQMXNiBz59XEpYAM62v6gQ== --osd-uuid 780522fb-cdae-4eb2-80e8-854a9ba99cbb --osd-objectstore=filestore 2021-07-26T06:40:26.194 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:26.195+0000 7fb8c6c320c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:26.207 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:26.208+0000 7fb8c6c320c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:26.208 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:26.209+0000 7fb8c6c320c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:26.819 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:26.820+0000 7fb8c6c320c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:40:27.339 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:27.340+0000 7fb8c6c320c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:40:27.339 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:27.340+0000 7fb8c6c320c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:40:27.339 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:27.340+0000 7fb8c6c320c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:40:28.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T06:40:28.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:40:28.220 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:40:28.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd2 key to auth repository 2021-07-26T06:40:28.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:40:28.560 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:40:28.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.2 2021-07-26T06:40:28.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:40:28.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:40:28.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:40:28.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:40:28.582 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:28.584+0000 7f45b1f5e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:28.584 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:28.585+0000 7f45b1f5e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:28.585 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:28.586+0000 7f45b1f5e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:28.586 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:28.587+0000 7f45b1f5e0c0 -1 Falling back to public interface 2021-07-26T06:40:29.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 2 2021-07-26T06:40:29.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:40:29.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:40:29.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:40:29.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:40:29.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:29.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:40:29.002 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:40:29.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:29.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:40:29.163 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:29.164+0000 7f45b1f5e0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:40:29.179 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:29.181+0000 7f45b1f5e0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:40:29.180 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:29.181+0000 7f45b1f5e0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:40:29.187 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:29.188+0000 7f45b1f5e0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:40:29.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:40:30.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:40:30.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:30.287 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:40:30.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:40:30.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:30.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:40:30.620 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/202942,v1:127.0.0.1:6819/202942] [v2:127.0.0.1:6820/202942,v1:127.0.0.1:6821/202942] exists,up 780522fb-cdae-4eb2-80e8-854a9ba99cbb 2021-07-26T06:40:30.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:40:30.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:40:30.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:40:30.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2021-07-26T06:40:30.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: '[' false = true ']' 2021-07-26T06:40:30.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd_filestore td/osd-scrub-repair 3 2021-07-26T06:40:30.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:40:30.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:40:30.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=3 2021-07-26T06:40:30.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:40:30.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:40:30.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:40:30.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:40:30.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:40:30.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:40:30.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:40:30.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:40:30.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:40:30.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:40:30.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:40:30.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:40:30.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:40:30.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:40:30.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:40:30.631 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:40:30.632 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:40:30.632 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:40:30.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:30.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:30.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:40:30.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:40:30.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:40:30.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:40:30.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:40:30.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:40:30.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:40:30.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:40:30.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:40:30.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:40:30.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:40:30.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:40:30.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=8fb4ec75-d2c4-4729-b80a-1796fd4c97f9 2021-07-26T06:40:30.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd2 8fb4ec75-d2c4-4729-b80a-1796fd4c97f9' 2021-07-26T06:40:30.636 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 8fb4ec75-d2c4-4729-b80a-1796fd4c97f9 2021-07-26T06:40:30.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:40:30.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQDeWP5gX4kYJhAAVXKUoRnuL3BLS39BaN2fFg== 2021-07-26T06:40:30.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQDeWP5gX4kYJhAAVXKUoRnuL3BLS39BaN2fFg=="}' 2021-07-26T06:40:30.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 8fb4ec75-d2c4-4729-b80a-1796fd4c97f9 -i td/osd-scrub-repair/3/new.json 2021-07-26T06:40:31.038 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:40:31.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/3/new.json 2021-07-26T06:40:31.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDeWP5gX4kYJhAAVXKUoRnuL3BLS39BaN2fFg== --osd-uuid 8fb4ec75-d2c4-4729-b80a-1796fd4c97f9 --osd-objectstore=filestore 2021-07-26T06:40:31.075 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:31.076+0000 7fef94f810c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:31.089 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:31.089+0000 7fef94f810c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:31.089 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:31.090+0000 7fef94f810c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:31.906 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:31.907+0000 7fef94f810c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:40:32.815 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:32.816+0000 7fef94f810c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:40:32.815 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:32.816+0000 7fef94f810c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:40:32.815 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:32.816+0000 7fef94f810c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:40:33.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/3/keyring 2021-07-26T06:40:33.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:40:33.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd3 key to auth repository 2021-07-26T06:40:33.383 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd3 key to auth repository 2021-07-26T06:40:33.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:40:33.732 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.3 not found in provided keyring 2021-07-26T06:40:33.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.3 2021-07-26T06:40:33.734 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.3 2021-07-26T06:40:33.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:40:33.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:40:33.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:40:33.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:40:33.756 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:33.757+0000 7f44364740c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:33.758 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:33.759+0000 7f44364740c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:33.758 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:33.760+0000 7f44364740c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:33.760 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:33.761+0000 7f44364740c0 -1 Falling back to public interface 2021-07-26T06:40:34.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 3 2021-07-26T06:40:34.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:40:34.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:40:34.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:40:34.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:40:34.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:34.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:40:34.176 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:40:34.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:34.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:40:34.497 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:34.497+0000 7f44364740c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:40:34.497 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:34.498+0000 7f44364740c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:40:34.498 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:34.498+0000 7f44364740c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:40:34.503 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:34.504+0000 7f44364740c0 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T06:40:34.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:40:35.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:40:35.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:35.512 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:40:35.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:40:35.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:35.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:40:35.888 INFO:tasks.workunit.client.0.smithi071.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/203225,v1:127.0.0.1:6827/203225] [v2:127.0.0.1:6828/203225,v1:127.0.0.1:6829/203225] exists,up 8fb4ec75-d2c4-4729-b80a-1796fd4c97f9 2021-07-26T06:40:35.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:40:35.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:40:35.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:40:35.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: create_rbd_pool 2021-07-26T06:40:35.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:40:36.214 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T06:40:36.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:40:36.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:40:36.616 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T06:40:36.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:40:37.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:40:40.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:809: corrupt_and_repair_jerasure: wait_for_clean 2021-07-26T06:40:40.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:40:40.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:40:40.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:40:40.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:40:40.295 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:40:40.295 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:40:40.295 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:40:40.296 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:40:40.296 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:40:40.297 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:40:40.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:40:40.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:40:40.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:40:40.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:40:40.402 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:40:40.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:40:40.669 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:40:40.670 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:40:40.670 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T06:40:40.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:40:40.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:40.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:40:40.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836485 2021-07-26T06:40:40.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836485 2021-07-26T06:40:40.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836485' 2021-07-26T06:40:40.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:40.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:40:40.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672965 2021-07-26T06:40:40.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672965 2021-07-26T06:40:40.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836485 1-42949672965' 2021-07-26T06:40:40.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:40.959 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:40:41.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509444 2021-07-26T06:40:41.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509444 2021-07-26T06:40:41.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836485 1-42949672965 2-64424509444' 2021-07-26T06:40:41.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:41.076 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:40:41.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345923 2021-07-26T06:40:41.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345923 2021-07-26T06:40:41.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836485 1-42949672965 2-64424509444 3-85899345923' 2021-07-26T06:40:41.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:41.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836485 2021-07-26T06:40:41.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:41.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:40:41.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836485 2021-07-26T06:40:41.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:41.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836485 2021-07-26T06:40:41.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2021-07-26T06:40:41.196 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836485 2021-07-26T06:40:41.197 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:41.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836484 -lt 21474836485 2021-07-26T06:40:41.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:40:42.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:40:42.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:42.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836485 -lt 21474836485 2021-07-26T06:40:42.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:42.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672965 2021-07-26T06:40:42.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:42.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:40:42.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672965 2021-07-26T06:40:42.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:42.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672965 2021-07-26T06:40:42.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-07-26T06:40:42.746 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672965 2021-07-26T06:40:42.747 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:40:43.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672965 -lt 42949672965 2021-07-26T06:40:43.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:43.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509444 2021-07-26T06:40:43.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:43.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:40:43.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509444 2021-07-26T06:40:43.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:43.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509444 2021-07-26T06:40:43.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-07-26T06:40:43.037 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509444 2021-07-26T06:40:43.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:40:43.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509445 -lt 64424509444 2021-07-26T06:40:43.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:43.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345923 2021-07-26T06:40:43.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:43.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:40:43.347 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345923 2021-07-26T06:40:43.348 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:43.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345923 2021-07-26T06:40:43.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345923' 2021-07-26T06:40:43.349 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.3 seq 85899345923 2021-07-26T06:40:43.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:40:43.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345924 -lt 85899345923 2021-07-26T06:40:43.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:40:43.657 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:40:43.657 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:40:43.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:40:43.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:40:43.982 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:40:43.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:40:43.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:40:43.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:40:43.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:40:43.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:40:44.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:40:44.283 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:40:44.283 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:40:44.283 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:40:44.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:40:44.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:40:44.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:40:44.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2021-07-26T06:40:44.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=ecpool 2021-07-26T06:40:44.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T06:40:44.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=false 2021-07-26T06:40:44.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T06:40:44.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2021-07-26T06:40:45.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2021-07-26T06:40:45.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2021-07-26T06:40:46.103 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' created 2021-07-26T06:40:46.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:40:47.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' false = true ']' 2021-07-26T06:40:47.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T06:40:47.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:40:47.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:40:47.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:40:47.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:40:47.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:40:47.117 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:40:47.117 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:40:47.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:40:47.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:40:47.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:40:47.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:40:47.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:40:47.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:40:47.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:40:47.222 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:40:47.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:40:47.519 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:40:47.520 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:40:47.520 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T06:40:47.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:40:47.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:47.521 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:40:47.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836488 2021-07-26T06:40:47.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836488 2021-07-26T06:40:47.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488' 2021-07-26T06:40:47.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:47.638 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:40:47.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672967 2021-07-26T06:40:47.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672967 2021-07-26T06:40:47.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488 1-42949672967' 2021-07-26T06:40:47.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:47.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:40:47.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509447 2021-07-26T06:40:47.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509447 2021-07-26T06:40:47.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-64424509447' 2021-07-26T06:40:47.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:47.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:40:48.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345926 2021-07-26T06:40:48.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345926 2021-07-26T06:40:48.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-64424509447 3-85899345926' 2021-07-26T06:40:48.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:48.113 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836488 2021-07-26T06:40:48.114 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:48.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:40:48.116 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836488 2021-07-26T06:40:48.116 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:48.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836488 2021-07-26T06:40:48.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-07-26T06:40:48.117 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836488 2021-07-26T06:40:48.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:48.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836488 2021-07-26T06:40:48.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:40:49.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:40:49.384 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:49.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836488 -lt 21474836488 2021-07-26T06:40:49.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:49.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672967 2021-07-26T06:40:49.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:49.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:40:49.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672967 2021-07-26T06:40:49.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:49.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672967 2021-07-26T06:40:49.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2021-07-26T06:40:49.695 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672967 2021-07-26T06:40:49.695 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:40:50.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672967 -lt 42949672967 2021-07-26T06:40:50.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:50.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509447 2021-07-26T06:40:50.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:50.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:40:50.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509447 2021-07-26T06:40:50.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:50.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509447 2021-07-26T06:40:50.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2021-07-26T06:40:50.028 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509447 2021-07-26T06:40:50.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:40:50.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509447 -lt 64424509447 2021-07-26T06:40:50.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:50.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345926 2021-07-26T06:40:50.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:50.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:40:50.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345926 2021-07-26T06:40:50.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:50.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345926 2021-07-26T06:40:50.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2021-07-26T06:40:50.445 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.3 seq 85899345926 2021-07-26T06:40:50.446 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:40:50.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345926 -lt 85899345926 2021-07-26T06:40:50.793 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:40:50.794 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:40:50.794 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:40:51.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:40:51.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:40:51.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:40:51.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:40:51.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:40:51.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:40:51.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:40:51.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:40:51.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:40:51.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:40:51.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:40:51.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:40:51.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:40:51.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:40:51.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:40:51.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T06:40:51.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:812: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2021-07-26T06:40:51.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2021-07-26T06:40:51.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local poolname=ecpool 2021-07-26T06:40:51.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:250: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2021-07-26T06:40:51.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:40:51.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T06:40:51.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2021-07-26T06:40:51.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:40:51.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:40:51.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:40:52.206 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:40:52.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:40:52.800 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:40:52.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:40:52.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:40:52.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T06:40:53.010 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2021-07-26T06:40:53.010 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=ecpool 2021-07-26T06:40:53.011 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:40:53.011 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:40:53.011 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:40:53.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: local primary=3 2021-07-26T06:40:53.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2021-07-26T06:40:53.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2021-07-26T06:40:53.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: sed -e s/3// 2021-07-26T06:40:53.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=ecpool 2021-07-26T06:40:53.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=SOMETHING 2021-07-26T06:40:53.282 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:40:53.282 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T06:40:53.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=3 2021-07-26T06:40:53.550 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:40:53.550 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:40:53.551 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T06:40:53.551 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 3 1 2 0 2021-07-26T06:40:53.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local -a osds 2021-07-26T06:40:53.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local not_primary_first=1 2021-07-26T06:40:53.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_second=2 2021-07-26T06:40:53.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2021-07-26T06:40:53.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2021-07-26T06:40:53.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2021-07-26T06:40:53.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=3 2021-07-26T06:40:53.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:40:53.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:40:53.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:40:53.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:40:53.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:40:53.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:40:53.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:40:53.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:40:53.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:40:53.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:40:53.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:40:53.558 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:40:53.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:40:53.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:40:53.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:40:53.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:40:53.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:40:53.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:40:53.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:40:53.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:40:53.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:40:53.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:40:53.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:40:53.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2021-07-26T06:40:54.608 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:40:54.609 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:40:54.609 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:40:54.609 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2021-07-26T06:40:55.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:40:55.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:40:55.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:40:55.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:40:55.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:40:55.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:40:55.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:40:55.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:40:55.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:40:55.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:40:55.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:40:55.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:40:55.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:40:55.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:40:55.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:40:55.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:40:55.283 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:40:55.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:40:55.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:40:55.284 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:40:55.284 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:40:55.285 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:40:55.285 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:55.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:40:55.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:40:55.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:40:55.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:40:55.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:40:55.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:40:55.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:40:55.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:40:55.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:40:55.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:40:55.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:40:55.287 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.3 2021-07-26T06:40:55.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:40:55.288 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:40:55.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:40:55.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:40:55.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:40:55.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:40:55.303 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:55.303+0000 7faa8913c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:55.328 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:55.328+0000 7faa8913c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:55.329 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:55.329+0000 7faa8913c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:55.330 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:55.331+0000 7faa8913c0c0 -1 Falling back to public interface 2021-07-26T06:40:55.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:40:55.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:40:55.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:40:55.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:40:55.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:40:55.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:55.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:40:55.563 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:40:55.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:55.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:40:56.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:40:56.142 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:56.142+0000 7faa8913c0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:40:56.146 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:56.147+0000 7faa8913c0c0 -1 journal do_read_entry(294912): bad header magic 2021-07-26T06:40:56.146 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:56.147+0000 7faa8913c0c0 -1 journal do_read_entry(294912): bad header magic 2021-07-26T06:40:56.153 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:40:56.153+0000 7faa8913c0c0 -1 osd.3 36 log_to_monitors {default=true} 2021-07-26T06:40:57.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:40:57.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:57.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:40:57.069 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:40:57.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:57.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:40:57.450 INFO:tasks.workunit.client.0.smithi071.stderr:osd.3 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/204548,v1:127.0.0.1:6827/204548] [v2:127.0.0.1:6828/204548,v1:127.0.0.1:6829/204548] exists,up 8fb4ec75-d2c4-4729-b80a-1796fd4c97f9 2021-07-26T06:40:57.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:40:57.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:40:57.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:40:57.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:40:57.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:40:57.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:40:57.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:40:57.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:40:57.453 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:40:57.453 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:40:57.453 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:40:57.453 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:40:57.454 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:40:57.454 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:40:57.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:40:57.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:40:57.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:40:57.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:40:57.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:40:57.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:40:57.863 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:40:57.863 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:40:57.863 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T06:40:57.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:40:57.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:57.864 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:40:57.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836491 2021-07-26T06:40:57.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836491 2021-07-26T06:40:57.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491' 2021-07-26T06:40:57.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:57.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:40:58.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672970 2021-07-26T06:40:58.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672970 2021-07-26T06:40:58.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2021-07-26T06:40:58.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:58.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:40:58.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509450 2021-07-26T06:40:58.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509450 2021-07-26T06:40:58.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509450' 2021-07-26T06:40:58.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:58.218 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:40:58.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691842 2021-07-26T06:40:58.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691842 2021-07-26T06:40:58.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509450 3-171798691842' 2021-07-26T06:40:58.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:58.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836491 2021-07-26T06:40:58.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:58.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:40:58.338 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836491 2021-07-26T06:40:58.339 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:58.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836491 2021-07-26T06:40:58.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-07-26T06:40:58.340 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836491 2021-07-26T06:40:58.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:58.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836490 -lt 21474836491 2021-07-26T06:40:58.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:40:59.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:40:59.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:59.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836491 -lt 21474836491 2021-07-26T06:40:59.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:59.910 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672970 2021-07-26T06:40:59.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:59.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:40:59.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672970 2021-07-26T06:40:59.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:59.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672970 2021-07-26T06:40:59.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2021-07-26T06:40:59.913 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672970 2021-07-26T06:40:59.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:41:00.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672970 -lt 42949672970 2021-07-26T06:41:00.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:00.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509450 2021-07-26T06:41:00.178 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:00.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:41:00.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509450 2021-07-26T06:41:00.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:00.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509450 2021-07-26T06:41:00.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2021-07-26T06:41:00.182 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509450 2021-07-26T06:41:00.182 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:41:00.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509450 -lt 64424509450 2021-07-26T06:41:00.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:00.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-171798691842 2021-07-26T06:41:00.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:00.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:41:00.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-171798691842 2021-07-26T06:41:00.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:00.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691842 2021-07-26T06:41:00.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 171798691842' 2021-07-26T06:41:00.529 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 171798691842 2021-07-26T06:41:00.530 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:41:00.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691842 -lt 171798691842 2021-07-26T06:41:00.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:41:00.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:41:00.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:41:01.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:41:01.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:41:01.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:41:01.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:41:01.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:41:01.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:41:01.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:41:01.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:41:01.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:41:01.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:41:01.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:41:01.659 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:41:01.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:41:01.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:41:01.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:41:01.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2021-07-26T06:41:01.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:41:01.985 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:41:01.985 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:41:01.985 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:41:02.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2021-07-26T06:41:02.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2021-07-26T06:41:02.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:41:02.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:41:02.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:02.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:02.295 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:02.295 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:02.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:40:45.783108+0000 2021-07-26T06:41:02.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:41:02.805 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:41:02.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:40:45.783108+0000 2021-07-26T06:41:02.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:41:02.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:40:45.783108+0000 2021-07-26T06:41:02.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:41:02.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:41:02.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:02.817 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:02.817 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:02.817 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:02.817 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:02.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:03.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:40:45.783108+0000 '>' 2021-07-26T06:40:45.783108+0000 2021-07-26T06:41:03.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:04.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:04.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:04.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:04.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:04.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:04.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:04.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:04.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:40:45.783108+0000 '>' 2021-07-26T06:40:45.783108+0000 2021-07-26T06:41:04.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:05.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:05.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:05.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:05.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:05.361 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:05.361 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:05.361 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:05.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:40:45.783108+0000 '>' 2021-07-26T06:40:45.783108+0000 2021-07-26T06:41:05.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:06.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:06.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:06.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:06.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:06.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:06.637 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:06.637 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:06.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:40:45.783108+0000 '>' 2021-07-26T06:40:45.783108+0000 2021-07-26T06:41:06.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:07.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:07.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:07.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:07.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:07.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:07.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:07.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:08.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:40:45.783108+0000 '>' 2021-07-26T06:40:45.783108+0000 2021-07-26T06:41:08.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:09.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:09.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:09.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:09.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:09.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:09.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:09.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:09.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:40:45.783108+0000 '>' 2021-07-26T06:40:45.783108+0000 2021-07-26T06:41:09.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:10.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:10.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:10.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:10.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:10.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:10.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:10.450 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:10.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:40:45.783108+0000 '>' 2021-07-26T06:40:45.783108+0000 2021-07-26T06:41:10.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:11.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:11.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:11.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:11.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:11.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:11.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:11.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:11.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:40:45.783108+0000 '>' 2021-07-26T06:40:45.783108+0000 2021-07-26T06:41:11.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:12.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:12.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:12.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:12.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:12.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:12.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:12.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:13.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:40:45.783108+0000 '>' 2021-07-26T06:40:45.783108+0000 2021-07-26T06:41:13.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:14.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:14.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:14.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:14.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:14.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:14.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:14.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:14.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:41:07.169310+0000 '>' 2021-07-26T06:40:45.783108+0000 2021-07-26T06:41:14.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:41:14.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:41:14.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:41:14.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:41:14.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:41:14.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:41:14.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:41:14.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:41:14.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:41:14.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:41:14.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:41:14.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:41:14.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:41:14.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:41:14.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:41:14.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:41:14.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:41:14.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:41:14.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:41:14.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:41:14.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:41:14.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:41:14.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:41:14.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:41:14.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2021-07-26T06:41:15.516 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:41:15.516 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:41:15.516 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2021-07-26T06:41:15.516 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:41:15.517 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:41:15.517 INFO:tasks.workunit.client.0.smithi071.stdout:hinfo_key 2021-07-26T06:41:15.517 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:41:15.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:41:15.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:41:15.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:41:15.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:41:15.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:41:15.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:41:15.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:41:15.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:41:15.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:41:15.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:41:15.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:41:15.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:41:15.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:41:15.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:41:15.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:41:15.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:41:15.692 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:41:15.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:41:15.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:41:15.693 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:41:15.693 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:41:15.694 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:41:15.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:41:15.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:41:15.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:41:15.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:41:15.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:41:15.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:41:15.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:41:15.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:41:15.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:41:15.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:41:15.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:41:15.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:41:15.697 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.3 2021-07-26T06:41:15.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:41:15.697 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:41:15.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:41:15.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:41:15.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:41:15.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:41:15.709 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:15.710+0000 7f13739de0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:41:15.711 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:15.712+0000 7f13739de0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:41:15.711 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:15.713+0000 7f13739de0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:41:15.713 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:15.714+0000 7f13739de0c0 -1 Falling back to public interface 2021-07-26T06:41:16.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:41:16.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:41:16.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:41:16.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:41:16.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:41:16.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:41:16.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:41:16.174 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:41:16.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:41:16.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:41:16.411 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:16.412+0000 7f13739de0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:41:16.419 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:16.420+0000 7f13739de0c0 -1 journal do_read_entry(438272): bad header magic 2021-07-26T06:41:16.420 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:16.420+0000 7f13739de0c0 -1 journal do_read_entry(438272): bad header magic 2021-07-26T06:41:16.427 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:16.428+0000 7f13739de0c0 -1 osd.3 42 log_to_monitors {default=true} 2021-07-26T06:41:16.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:41:17.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:41:17.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:41:17.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:41:17.501 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:41:17.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:41:17.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:41:18.009 INFO:tasks.workunit.client.0.smithi071.stderr:osd.3 up in weight 1 up_from 46 up_thru 46 down_at 43 last_clean_interval [40,42) [v2:127.0.0.1:6826/205536,v1:127.0.0.1:6827/205536] [v2:127.0.0.1:6828/205536,v1:127.0.0.1:6829/205536] exists,up 8fb4ec75-d2c4-4729-b80a-1796fd4c97f9 2021-07-26T06:41:18.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:41:18.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:41:18.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:41:18.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:41:18.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:41:18.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:41:18.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:41:18.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:41:18.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:41:18.012 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:41:18.012 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:41:18.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:41:18.013 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:41:18.013 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:41:18.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:41:18.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:41:18.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:41:18.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:41:18.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:41:18.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:41:18.469 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:41:18.469 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:41:18.469 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T06:41:18.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:41:18.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:41:18.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:41:18.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836496 2021-07-26T06:41:18.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836496 2021-07-26T06:41:18.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496' 2021-07-26T06:41:18.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:41:18.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:41:18.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672975 2021-07-26T06:41:18.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672975 2021-07-26T06:41:18.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672975' 2021-07-26T06:41:18.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:41:18.704 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:41:18.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509455 2021-07-26T06:41:18.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509455 2021-07-26T06:41:18.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-64424509455' 2021-07-26T06:41:18.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:41:18.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:41:18.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=197568495618 2021-07-26T06:41:18.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 197568495618 2021-07-26T06:41:18.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-64424509455 3-197568495618' 2021-07-26T06:41:18.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:18.938 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836496 2021-07-26T06:41:18.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:18.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:41:18.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836496 2021-07-26T06:41:18.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:18.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836496 2021-07-26T06:41:18.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-07-26T06:41:18.941 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836496 2021-07-26T06:41:18.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:41:19.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836495 -lt 21474836496 2021-07-26T06:41:19.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:41:20.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:41:20.207 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:41:20.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836496 -lt 21474836496 2021-07-26T06:41:20.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:20.476 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672975 2021-07-26T06:41:20.476 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:20.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:41:20.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672975 2021-07-26T06:41:20.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:20.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672975 2021-07-26T06:41:20.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2021-07-26T06:41:20.479 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672975 2021-07-26T06:41:20.479 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:41:20.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672975 -lt 42949672975 2021-07-26T06:41:20.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:20.786 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509455 2021-07-26T06:41:20.787 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:20.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:41:20.788 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509455 2021-07-26T06:41:20.788 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:20.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509455 2021-07-26T06:41:20.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2021-07-26T06:41:20.789 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509455 2021-07-26T06:41:20.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:41:21.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509455 -lt 64424509455 2021-07-26T06:41:21.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:21.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-197568495618 2021-07-26T06:41:21.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:21.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:41:21.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-197568495618 2021-07-26T06:41:21.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:21.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=197568495618 2021-07-26T06:41:21.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2021-07-26T06:41:21.102 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 197568495618 2021-07-26T06:41:21.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:41:21.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 197568495618 -lt 197568495618 2021-07-26T06:41:21.370 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:41:21.371 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:41:21.371 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:41:21.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:41:21.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:41:21.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:41:21.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:41:21.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:41:21.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:41:21.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:41:21.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:41:22.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:41:22.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:41:22.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:41:22.181 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:41:22.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:41:22.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:41:22.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:41:22.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:41:31.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:41:31.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2021-07-26T06:41:31.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2021-07-26T06:41:31.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2021-07-26T06:41:31.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=1 2021-07-26T06:41:31.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:41:31.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:41:31.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:41:31.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:41:31.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:41:31.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:41:31.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:41:31.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:41:31.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:41:31.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:41:31.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:41:31.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:41:31.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:41:31.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:41:31.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:41:31.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:41:31.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:41:31.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:41:31.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:41:31.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:41:31.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:41:31.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:41:31.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:41:31.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2021-07-26T06:41:32.977 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:41:32.978 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:41:32.978 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:41:32.978 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:41:32.979 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:41:33.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:41:33.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:41:33.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:41:33.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:41:33.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:41:33.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:41:33.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:41:33.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:41:33.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:41:33.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:41:33.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:41:33.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:41:33.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:41:33.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:41:33.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:41:33.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:41:33.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:41:33.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:41:33.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:41:33.582 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:41:33.583 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:41:33.583 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:41:33.583 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:41:33.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:41:33.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:41:33.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:41:33.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:41:33.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:41:33.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:41:33.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:41:33.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:41:33.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:41:33.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:41:33.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:41:33.586 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:41:33.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:41:33.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:41:33.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:41:33.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:41:33.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:41:33.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:41:33.601 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:33.602+0000 7f8c140d20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:41:33.688 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:33.689+0000 7f8c140d20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:41:33.689 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:33.690+0000 7f8c140d20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:41:33.690 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:33.691+0000 7f8c140d20c0 -1 Falling back to public interface 2021-07-26T06:41:33.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:41:33.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:41:33.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:41:33.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:41:33.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:41:33.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:41:33.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:41:33.995 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:41:33.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:41:33.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:41:34.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:41:34.383 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:34.384+0000 7f8c140d20c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:41:34.383 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:34.384+0000 7f8c140d20c0 -1 journal do_read_entry(561152): bad header magic 2021-07-26T06:41:34.384 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:34.384+0000 7f8c140d20c0 -1 journal do_read_entry(561152): bad header magic 2021-07-26T06:41:34.391 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:34.392+0000 7f8c140d20c0 -1 osd.1 48 log_to_monitors {default=true} 2021-07-26T06:41:35.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:41:35.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:41:35.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:41:35.357 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:41:35.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:41:35.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:41:35.796 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 49 last_clean_interval [10,48) [v2:127.0.0.1:6810/206239,v1:127.0.0.1:6811/206239] [v2:127.0.0.1:6812/206239,v1:127.0.0.1:6813/206239] exists,up 31c57fb5-51ff-4cb0-a76a-87197652a01e 2021-07-26T06:41:35.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:41:35.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:41:35.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:41:35.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:41:35.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:41:35.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:41:35.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:41:35.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:41:35.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:41:35.798 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:41:35.799 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:41:35.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:41:35.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:41:35.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:41:35.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:41:35.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:41:35.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:41:35.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:41:35.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:41:36.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:41:36.158 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:41:36.159 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:41:36.159 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T06:41:36.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:41:36.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:41:36.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:41:36.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836500 2021-07-26T06:41:36.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836500 2021-07-26T06:41:36.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836500' 2021-07-26T06:41:36.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:41:36.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:41:36.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=227633266690 2021-07-26T06:41:36.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 227633266690 2021-07-26T06:41:36.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836500 1-227633266690' 2021-07-26T06:41:36.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:41:36.466 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:41:36.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509460 2021-07-26T06:41:36.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509460 2021-07-26T06:41:36.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836500 1-227633266690 2-64424509460' 2021-07-26T06:41:36.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:41:36.583 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:41:36.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=197568495622 2021-07-26T06:41:36.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 197568495622 2021-07-26T06:41:36.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836500 1-227633266690 2-64424509460 3-197568495622' 2021-07-26T06:41:36.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:36.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836500 2021-07-26T06:41:36.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:36.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:41:36.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836500 2021-07-26T06:41:36.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:36.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836500 2021-07-26T06:41:36.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2021-07-26T06:41:36.703 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836500 2021-07-26T06:41:36.703 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:41:36.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836499 -lt 21474836500 2021-07-26T06:41:36.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:41:37.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:41:37.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:41:38.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836501 -lt 21474836500 2021-07-26T06:41:38.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:38.247 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-227633266690 2021-07-26T06:41:38.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:38.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:41:38.249 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-227633266690 2021-07-26T06:41:38.249 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:38.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=227633266690 2021-07-26T06:41:38.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 227633266690' 2021-07-26T06:41:38.251 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 227633266690 2021-07-26T06:41:38.251 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:41:38.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 227633266690 -lt 227633266690 2021-07-26T06:41:38.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:38.558 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509460 2021-07-26T06:41:38.558 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:38.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:41:38.560 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509460 2021-07-26T06:41:38.560 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:38.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509460 2021-07-26T06:41:38.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2021-07-26T06:41:38.561 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509460 2021-07-26T06:41:38.562 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:41:38.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509460 -lt 64424509460 2021-07-26T06:41:38.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:38.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-197568495622 2021-07-26T06:41:38.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:38.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:41:38.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-197568495622 2021-07-26T06:41:38.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:38.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=197568495622 2021-07-26T06:41:38.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 197568495622' 2021-07-26T06:41:38.829 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 197568495622 2021-07-26T06:41:38.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:41:39.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 197568495623 -lt 197568495622 2021-07-26T06:41:39.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:41:39.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:41:39.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:41:39.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:41:39.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:41:39.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:41:39.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:41:39.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:41:39.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:41:39.428 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:41:39.428 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:41:39.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:41:39.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:41:39.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:41:39.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:41:40.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:41:40.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:41:40.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:41:40.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2021-07-26T06:41:40.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:41:40.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:41:40.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:41:40.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:41:40.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2021-07-26T06:41:40.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2021-07-26T06:41:40.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:41:40.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:41:40.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:40.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:40.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:40.405 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:40.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:41:07.169310+0000 2021-07-26T06:41:40.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:41:40.958 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:41:40.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:41:07.169310+0000 2021-07-26T06:41:40.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:41:40.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:41:07.169310+0000 2021-07-26T06:41:40.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:41:40.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:41:40.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:40.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:40.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:40.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:40.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:40.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:41.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:41:07.169310+0000 '>' 2021-07-26T06:41:07.169310+0000 2021-07-26T06:41:41.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:42.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:42.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:42.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:42.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:42.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:42.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:42.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:42.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:41:07.169310+0000 '>' 2021-07-26T06:41:07.169310+0000 2021-07-26T06:41:42.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:43.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:43.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:43.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:43.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:43.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:43.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:43.537 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:43.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:41:07.169310+0000 '>' 2021-07-26T06:41:07.169310+0000 2021-07-26T06:41:43.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:44.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:44.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:44.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:44.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:44.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:44.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:44.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:45.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:41:07.169310+0000 '>' 2021-07-26T06:41:07.169310+0000 2021-07-26T06:41:45.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:46.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:46.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:46.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:46.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:46.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:46.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:46.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:46.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:41:07.169310+0000 '>' 2021-07-26T06:41:07.169310+0000 2021-07-26T06:41:46.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:41:47.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:41:47.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:41:47.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:41:47.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:41:47.359 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:41:47.359 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:41:47.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:41:47.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:41:42.144807+0000 '>' 2021-07-26T06:41:07.169310+0000 2021-07-26T06:41:47.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:41:47.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:41:47.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:41:47.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:41:47.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:41:47.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:41:47.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:41:47.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:41:47.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:41:47.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:41:47.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:41:47.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:41:47.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:41:47.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:41:47.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:41:47.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:41:47.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:41:47.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:41:47.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:41:47.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:41:47.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:41:47.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:41:47.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:41:47.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:41:47.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2021-07-26T06:41:48.521 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:41:48.521 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:41:48.522 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2021-07-26T06:41:48.522 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:41:48.522 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:41:48.523 INFO:tasks.workunit.client.0.smithi071.stdout:hinfo_key 2021-07-26T06:41:48.523 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:41:49.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:41:49.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:41:49.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:41:49.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:41:49.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:41:49.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:41:49.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:41:49.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:41:49.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:41:49.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:41:49.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:41:49.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:41:49.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:41:49.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:41:49.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:41:49.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:41:49.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:41:49.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:41:49.072 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:41:49.072 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:41:49.072 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:41:49.072 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:41:49.073 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:41:49.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:41:49.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:41:49.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:41:49.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:41:49.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:41:49.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:41:49.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:41:49.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:41:49.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:41:49.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:41:49.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:41:49.075 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.3 2021-07-26T06:41:49.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:41:49.076 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:41:49.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:41:49.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:41:49.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:41:49.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:41:49.091 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:49.091+0000 7f104c1110c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:41:49.110 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:49.110+0000 7f104c1110c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:41:49.111 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:49.112+0000 7f104c1110c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:41:49.113 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:49.113+0000 7f104c1110c0 -1 Falling back to public interface 2021-07-26T06:41:49.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:41:49.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:41:49.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:41:49.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:41:49.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:41:49.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:41:49.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:41:49.518 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:41:49.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:41:49.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:41:49.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:41:50.187 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:50.188+0000 7f104c1110c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:41:50.205 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:50.205+0000 7f104c1110c0 -1 journal do_read_entry(704512): bad header magic 2021-07-26T06:41:50.205 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:50.205+0000 7f104c1110c0 -1 journal do_read_entry(704512): bad header magic 2021-07-26T06:41:50.212 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:41:50.212+0000 7f104c1110c0 -1 osd.3 55 log_to_monitors {default=true} 2021-07-26T06:41:50.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:41:50.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:41:50.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:41:50.852 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:41:50.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:41:50.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:41:51.146 INFO:tasks.workunit.client.0.smithi071.stderr:osd.3 up in weight 1 up_from 59 up_thru 59 down_at 56 last_clean_interval [46,55) [v2:127.0.0.1:6826/207105,v1:127.0.0.1:6827/207105] [v2:127.0.0.1:6828/207105,v1:127.0.0.1:6829/207105] exists,up 8fb4ec75-d2c4-4729-b80a-1796fd4c97f9 2021-07-26T06:41:51.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:41:51.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:41:51.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:41:51.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:41:51.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:41:51.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:41:51.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:41:51.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:41:51.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:41:51.149 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:41:51.149 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:41:51.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:41:51.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:41:51.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:41:51.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:41:51.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:41:51.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:41:51.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:41:51.243 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:41:51.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:41:51.511 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:41:51.512 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:41:51.512 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T06:41:51.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:41:51.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:41:51.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:41:51.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836504 2021-07-26T06:41:51.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836504 2021-07-26T06:41:51.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836504' 2021-07-26T06:41:51.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:41:51.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:41:51.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=227633266694 2021-07-26T06:41:51.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 227633266694 2021-07-26T06:41:51.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836504 1-227633266694' 2021-07-26T06:41:51.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:41:51.747 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:41:51.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509464 2021-07-26T06:41:51.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509464 2021-07-26T06:41:51.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836504 1-227633266694 2-64424509464' 2021-07-26T06:41:51.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:41:51.886 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:41:52.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=253403070466 2021-07-26T06:41:52.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 253403070466 2021-07-26T06:41:52.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836504 1-227633266694 2-64424509464 3-253403070466' 2021-07-26T06:41:52.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:52.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836504 2021-07-26T06:41:52.004 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:52.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:41:52.006 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836504 2021-07-26T06:41:52.006 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:52.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836504 2021-07-26T06:41:52.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2021-07-26T06:41:52.007 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836504 2021-07-26T06:41:52.007 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:41:52.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836503 -lt 21474836504 2021-07-26T06:41:52.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:41:53.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:41:53.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:41:53.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836503 -lt 21474836504 2021-07-26T06:41:53.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:41:54.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:41:54.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:41:54.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836505 -lt 21474836504 2021-07-26T06:41:54.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:54.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-227633266694 2021-07-26T06:41:54.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:54.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:41:54.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-227633266694 2021-07-26T06:41:54.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:54.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=227633266694 2021-07-26T06:41:54.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 227633266694' 2021-07-26T06:41:54.828 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 227633266694 2021-07-26T06:41:54.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:41:55.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 227633266694 -lt 227633266694 2021-07-26T06:41:55.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:55.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509464 2021-07-26T06:41:55.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:55.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:41:55.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509464 2021-07-26T06:41:55.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:55.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509464 2021-07-26T06:41:55.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2021-07-26T06:41:55.109 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509464 2021-07-26T06:41:55.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:41:55.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509464 -lt 64424509464 2021-07-26T06:41:55.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:41:55.491 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-253403070466 2021-07-26T06:41:55.492 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:41:55.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:41:55.493 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-253403070466 2021-07-26T06:41:55.493 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:41:55.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=253403070466 2021-07-26T06:41:55.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 253403070466' 2021-07-26T06:41:55.494 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 253403070466 2021-07-26T06:41:55.494 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:41:55.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 253403070466 -lt 253403070466 2021-07-26T06:41:55.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:41:55.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:41:55.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:41:56.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:41:56.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:41:56.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:41:56.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:41:56.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:41:56.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:41:56.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:41:56.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:41:56.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:41:56.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:41:56.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:41:56.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:41:56.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:41:56.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:41:56.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:41:56.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:42:05.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:42:05.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2021-07-26T06:42:05.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2021-07-26T06:42:05.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2021-07-26T06:42:05.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=1 2021-07-26T06:42:05.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=2 2021-07-26T06:42:05.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2021-07-26T06:42:05.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:42:05.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:42:05.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:42:05.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 207736"' 2021-07-26T06:42:05.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 207736' 2021-07-26T06:42:05.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T06:42:05.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:42:05.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:42:05.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:42:05.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 207738"' 2021-07-26T06:42:05.476 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 207738' 2021-07-26T06:42:05.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2021-07-26T06:42:05.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 207736 207738' 2021-07-26T06:42:05.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:42:05.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:42:05.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 207736 2021-07-26T06:42:05.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/207739: /' 2021-07-26T06:42:05.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T06:42:05.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/207741: /' 2021-07-26T06:42:07.180 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:42:07.180 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:42:07.180 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=2 2021-07-26T06:42:07.181 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:42:07.181 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T06:42:07.181 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:42:07.181 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:42:07.182 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=2 2021-07-26T06:42:07.182 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:42:07.182 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2021-07-26T06:42:07.182 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:42:07.183 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:42:07.183 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:42:07.183 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:42:07.183 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:42:07.184 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:42:07.184 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T06:42:07.184 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:42:07.184 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:42:07.184 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=2 2021-07-26T06:42:07.185 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:42:07.185 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:42:07.185 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2021-07-26T06:42:07.185 INFO:tasks.workunit.client.0.smithi071.stderr:207741: Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:42:07.186 INFO:tasks.workunit.client.0.smithi071.stderr:207741: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:42:07.186 INFO:tasks.workunit.client.0.smithi071.stderr:207741: Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:42:07.186 INFO:tasks.workunit.client.0.smithi071.stderr:207741: Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T06:42:07.186 INFO:tasks.workunit.client.0.smithi071.stderr:207741: remove 2#2:eb822e21:::SOMETHING:head# 2021-07-26T06:42:07.187 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2021-07-26T06:42:07.187 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:42:07.187 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:42:07.187 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:42:07.187 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:42:07.188 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:42:07.188 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:42:07.188 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:42:07.188 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:42:07.189 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:42:07.189 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:42:07.189 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:42:07.189 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:42:07.190 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:42:07.190 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:42:07.190 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:42:07.190 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:42:07.190 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:42:07.191 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:42:07.191 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:42:07.191 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:42:07.191 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:42:07.192 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:42:07.192 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:42:07.192 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:42:07.192 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:42:07.193 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:42:07.193 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:42:07.193 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2021-07-26T06:42:07.193 INFO:tasks.workunit.client.0.smithi071.stderr:207739: Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:42:07.194 INFO:tasks.workunit.client.0.smithi071.stderr:207739: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:42:07.194 INFO:tasks.workunit.client.0.smithi071.stderr:207739: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:42:07.194 INFO:tasks.workunit.client.0.smithi071.stderr:207739: Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:42:07.194 INFO:tasks.workunit.client.0.smithi071.stderr:207739: remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:42:07.195 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:42:07.195 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:42:07.195 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:42:07.195 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:42:07.195 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:42:07.196 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:42:07.196 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:42:07.197 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:42:07.197 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:42:07.197 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:42:07.197 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:42:07.198 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:42:07.198 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:42:07.198 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:42:07.198 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:42:07.199 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:42:07.199 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:42:07.199 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:42:07.199 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:42:07.199 INFO:tasks.workunit.client.0.smithi071.stderr:207741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:42:07.200 INFO:tasks.workunit.client.0.smithi071.stderr:207741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:07.200 INFO:tasks.workunit.client.0.smithi071.stderr:207741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:07.200 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:07.200 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:07.201 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:42:07.201 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:42:07.201 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:42:07.201 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:42:07.202 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:42:07.202 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:07.202 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:42:07.202 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:42:07.203 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:42:07.203 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:42:07.203 INFO:tasks.workunit.client.0.smithi071.stderr:207741: start osd.2 2021-07-26T06:42:07.203 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-diargs=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:42:07.203 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:42:07.204 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:42:07.204 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:42:07.204 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:42:07.204 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:42:07.205 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:42:07.205 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:42:07.205 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:42:07.205 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:42:07.205 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:42:07.206 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:42:07.206 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:42:07.206 INFO:tasks.workunit.client.0.smithi071.stderr:207739: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:42:07.206 INFO:tasks.workunit.client.0.smithi071.stderr:207739: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:07.207 INFO:tasks.workunit.client.0.smithi071.stderr:207739: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:07.207 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:07.207 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:07.207 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:42:07.208 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:42:07.208 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:42:07.208 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:42:07.208 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:42:07.208 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:07.209 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:42:07.209 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:42:07.209 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:42:07.209 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:42:07.210 INFO:tasks.workunit.client.0.smithi071.stderr:207739: start osd.1 2021-07-26T06:42:09.199 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:09.199 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:42:09.199 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:42:09.200 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:42:09.200 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:42:09.200 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:42:09.200 INFO:tasks.workunit.client.0.smithi071.stderr:207739: 2021-07-26T06:42:07.208+0000 7fade7f8d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:09.201 INFO:tasks.workunit.client.0.smithi071.stderr:207739: 2021-07-26T06:42:07.211+0000 7fade7f8d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:09.201 INFO:tasks.workunit.client.0.smithi071.stderr:207739: 2021-07-26T06:42:07.212+0000 7fade7f8d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:09.201 INFO:tasks.workunit.client.0.smithi071.stderr:207739: 2021-07-26T06:42:07.214+0000 7fade7f8d0c0 -1 Falling back to public interface 2021-07-26T06:42:09.201 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:42:09.202 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:42:09.202 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:42:09.202 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:42:09.202 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:42:09.203 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:09.203 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:42:09.203 INFO:tasks.workunit.client.0.smithi071.stderr:207739: 0 2021-07-26T06:42:09.203 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:09.204 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:42:09.204 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:42:09.204 INFO:tasks.workunit.client.0.smithi071.stderr:207739: 2021-07-26T06:42:08.029+0000 7fade7f8d0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:42:09.204 INFO:tasks.workunit.client.0.smithi071.stderr:207739: 2021-07-26T06:42:08.071+0000 7fade7f8d0c0 -1 journal do_read_entry(843776): bad header magic 2021-07-26T06:42:09.204 INFO:tasks.workunit.client.0.smithi071.stderr:207739: 2021-07-26T06:42:08.071+0000 7fade7f8d0c0 -1 journal do_read_entry(843776): bad header magic 2021-07-26T06:42:09.205 INFO:tasks.workunit.client.0.smithi071.stderr:207739: 2021-07-26T06:42:08.078+0000 7fade7f8d0c0 -1 osd.1 60 log_to_monitors {default=true} 2021-07-26T06:42:09.205 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:42:09.205 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:09.205 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:42:09.206 INFO:tasks.workunit.client.0.smithi071.stderr:207739: 1 2021-07-26T06:42:09.206 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:09.206 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:42:09.206 INFO:tasks.workunit.client.0.smithi071.stderr:207739: osd.1 up in weight 1 up_from 65 up_thru 65 down_at 61 last_clean_interval [53,60) [v2:127.0.0.1:6811/207882,v1:127.0.0.1:6813/207882] [v2:127.0.0.1:6815/207882,v1:127.0.0.1:6817/207882] exists,up 31c57fb5-51ff-4cb0-a76a-87197652a01e 2021-07-26T06:42:09.207 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:42:09.207 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:42:09.207 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:42:09.207 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:42:09.208 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /r=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:09.208 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/2/whoami 2021-07-26T06:42:09.208 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:42:09.208 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:42:09.209 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:42:09.209 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:42:09.209 INFO:tasks.workunit.client.0.smithi071.stderr:207741: 2021-07-26T06:42:07.208+0000 7f3d2197f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:09.209 INFO:tasks.workunit.client.0.smithi071.stderr:207741: 2021-07-26T06:42:07.211+0000 7f3d2197f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:09.209 INFO:tasks.workunit.client.0.smithi071.stderr:207741: 2021-07-26T06:42:07.212+0000 7f3d2197f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:09.210 INFO:tasks.workunit.client.0.smithi071.stderr:207741: 2021-07-26T06:42:07.214+0000 7f3d2197f0c0 -1 Falling back to public interface 2021-07-26T06:42:09.210 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:42:09.210 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:42:09.210 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:42:09.211 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:42:09.211 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:42:09.211 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:09.211 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:42:09.211 INFO:tasks.workunit.client.0.smithi071.stderr:207741: 0 2021-07-26T06:42:09.212 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:09.212 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:42:09.212 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:42:09.212 INFO:tasks.workunit.client.0.smithi071.stderr:207741: 2021-07-26T06:42:08.029+0000 7f3d2197f0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:42:09.212 INFO:tasks.workunit.client.0.smithi071.stderr:207741: 2021-07-26T06:42:08.071+0000 7f3d2197f0c0 -1 journal do_read_entry(774144): bad header magic 2021-07-26T06:42:09.213 INFO:tasks.workunit.client.0.smithi071.stderr:207741: 2021-07-26T06:42:08.071+0000 7f3d2197f0c0 -1 journal do_read_entry(774144): bad header magic 2021-07-26T06:42:09.213 INFO:tasks.workunit.client.0.smithi071.stderr:207741: 2021-07-26T06:42:08.078+0000 7f3d2197f0c0 -1 osd.2 60 log_to_monitors {default=true} 2021-07-26T06:42:09.213 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:42:09.213 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:09.214 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:42:09.214 INFO:tasks.workunit.client.0.smithi071.stderr:207741: 1 2021-07-26T06:42:09.214 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:09.214 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:42:09.215 INFO:tasks.workunit.client.0.smithi071.stderr:207741: osd.2 up in weight 1 up_from 65 up_thru 49 down_at 61 last_clean_interval [15,60) [v2:127.0.0.1:6810/207883,v1:127.0.0.1:6812/207883] [v2:127.0.0.1:6814/207883,v1:127.0.0.1:6816/207883] exists,up 780522fb-cdae-4eb2-80e8-854a9ba99cbb 2021-07-26T06:42:09.215 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:42:09.215 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:42:09.215 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:42:09.216 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:42:10.448 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:42:10.449 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:42:10.449 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:42:10.449 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:42:10.450 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:42:10.450 INFO:tasks.workunit.client.0.smithi071.stderr:207741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:42:10.450 INFO:tasks.workunit.client.0.smithi071.stderr:207741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:42:10.450 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:42:10.451 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:42:10.451 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:42:10.451 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:42:10.451 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:42:10.451 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:42:10.452 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:42:10.452 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:42:10.452 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:42:10.452 INFO:tasks.workunit.client.0.smithi071.stderr:207741: 1 2021-07-26T06:42:10.453 INFO:tasks.workunit.client.0.smithi071.stderr:207741: 2 2021-07-26T06:42:10.453 INFO:tasks.workunit.client.0.smithi071.stderr:207741: 3' 2021-07-26T06:42:10.453 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:42:10.453 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:10.454 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:42:10.454 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836509 2021-07-26T06:42:10.454 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836509 2021-07-26T06:42:10.454 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836509' 2021-07-26T06:42:10.455 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:10.455 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:42:10.455 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=279172874243 2021-07-26T06:42:10.455 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 279172874243 2021-07-26T06:42:10.455 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836509 1-279172874243' 2021-07-26T06:42:10.456 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:10.456 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:42:10.456 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=279172874242 2021-07-26T06:42:10.456 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 279172874242 2021-07-26T06:42:10.457 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836509 1-279172874243 2-279172874242' 2021-07-26T06:42:10.457 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:10.457 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_shome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:42:10.457 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:42:10.458 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:42:10.458 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:42:10.458 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:42:10.458 INFO:tasks.workunit.client.0.smithi071.stderr:207739: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:42:10.459 INFO:tasks.workunit.client.0.smithi071.stderr:207739: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:42:10.459 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:42:10.459 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:42:10.459 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:42:10.459 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:42:10.460 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:42:10.460 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:42:10.460 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:42:10.460 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:42:10.461 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:42:10.461 INFO:tasks.workunit.client.0.smithi071.stderr:207739: 1 2021-07-26T06:42:10.461 INFO:tasks.workunit.client.0.smithi071.stderr:207739: 2 2021-07-26T06:42:10.461 INFO:tasks.workunit.client.0.smithi071.stderr:207739: 3' 2021-07-26T06:42:10.462 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:42:10.462 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:10.462 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:42:10.463 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836510 2021-07-26T06:42:10.463 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836510 2021-07-26T06:42:10.463 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510' 2021-07-26T06:42:10.463 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:10.463 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:42:10.464 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=279172874242 2021-07-26T06:42:10.464 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 279172874242 2021-07-26T06:42:10.464 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510 1-279172874242' 2021-07-26T06:42:10.464 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:10.465 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:42:10.465 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=279172874243 2021-07-26T06:42:10.465 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 279172874243 2021-07-26T06:42:10.465 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510 1-279172874242 2-279172874243' 2021-07-26T06:42:10.466 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:12.488 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:42:12.488 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=253403070472 2021-07-26T06:42:12.488 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 253403070472 2021-07-26T06:42:12.489 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836509 1-279172874243 2-279172874242 3-253403070472' 2021-07-26T06:42:12.489 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:12.489 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836509 2021-07-26T06:42:12.489 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:12.489 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:42:12.490 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836509 2021-07-26T06:42:12.490 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:12.490 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836509 2021-07-26T06:42:12.490 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2021-07-26T06:42:12.491 INFO:tasks.workunit.client.0.smithi071.stderr:207741: waiting osd.0 seq 21474836509 2021-07-26T06:42:12.491 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:12.491 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836508 -lt 21474836509 2021-07-26T06:42:12.491 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:42:12.491 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:42:12.492 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:12.492 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836510 -lt 21474836509 2021-07-26T06:42:12.492 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:12.492 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-279172874243 2021-07-26T06:42:12.493 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:12.493 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:42:12.493 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-279172874243 2021-07-26T06:42:12.493 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:12.493 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=279172874243 2021-07-26T06:42:12.494 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 279172874243' 2021-07-26T06:42:12.494 INFO:tasks.workunit.client.0.smithi071.stderr:207741: waiting osd.1 seq 279172874243 2021-07-26T06:42:12.494 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:42:12.494 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 279172874243 -lt 279172874243 2021-07-26T06:42:12.495 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:12.495 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-279172874242 2021-07-26T06:42:12.495 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:12.495 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:42:12.495 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-279172874242 2021-07-26T06:42:12.496 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:12.496 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flustats 2021-07-26T06:42:12.497 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=253403070471 2021-07-26T06:42:12.497 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 253403070471 2021-07-26T06:42:12.497 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510 1-279172874242 2-279172874243 3-253403070471' 2021-07-26T06:42:12.497 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:12.497 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836510 2021-07-26T06:42:12.498 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:12.498 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:42:12.498 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836510 2021-07-26T06:42:12.498 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:12.498 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836510 2021-07-26T06:42:12.499 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2021-07-26T06:42:12.499 INFO:tasks.workunit.client.0.smithi071.stderr:207739: waiting osd.0 seq 21474836510 2021-07-26T06:42:12.499 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:12.499 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836508 -lt 21474836510 2021-07-26T06:42:12.500 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:42:12.500 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:42:12.500 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:12.500 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836510 -lt 21474836510 2021-07-26T06:42:12.501 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:12.501 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-279172874242 2021-07-26T06:42:12.501 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:12.501 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:42:12.501 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-279172874242 2021-07-26T06:42:12.502 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:12.502 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=279172874242 2021-07-26T06:42:12.502 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 279172874242' 2021-07-26T06:42:12.502 INFO:tasks.workunit.client.0.smithi071.stderr:207739: waiting osd.1 seq 279172874242 2021-07-26T06:42:12.502 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:42:12.503 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 279172874243 -lt 279172874242 2021-07-26T06:42:12.503 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:12.503 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-279172874243 2021-07-26T06:42:12.503 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:12.503 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:42:12.504 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-279172874243 2021-07-26T06:42:12.504 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:14.141 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=279172874242 2021-07-26T06:42:14.142 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 279172874242' 2021-07-26T06:42:14.142 INFO:tasks.workunit.client.0.smithi071.stderr:207741: waiting osd.2 seq 279172874242 2021-07-26T06:42:14.142 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:42:14.142 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 279172874243 -lt 279172874242 2021-07-26T06:42:14.143 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:14.143 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-253403070472 2021-07-26T06:42:14.143 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:14.143 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:42:14.144 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-253403070472 2021-07-26T06:42:14.144 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:14.144 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=253403070472 2021-07-26T06:42:14.144 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 253403070472' 2021-07-26T06:42:14.145 INFO:tasks.workunit.client.0.smithi071.stderr:207741: waiting osd.3 seq 253403070472 2021-07-26T06:42:14.145 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:42:14.145 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 253403070472 -lt 253403070472 2021-07-26T06:42:14.145 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:42:14.146 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:14.146 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:14.146 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:42:14.147 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:42:14.147 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:42:14.147 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:42:14.147 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:42:14.148 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:42:14.148 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:42:14.148 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:42:14.148 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:42:14.148 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:42:14.149 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:14.149 INFO:tasks.workunit.client.0.smithi071.stderr:207741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:14.149 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:42:14.149 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:42:14.150 INFO:tasks.workunit.client.0.smithi071.stderr:207741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:42:14.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:42:14.152 INFO:tasks.workunit.client.0.smithi071.stderr:h_pg_stats: seq=279172874243 2021-07-26T06:42:14.152 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 279172874243' 2021-07-26T06:42:14.152 INFO:tasks.workunit.client.0.smithi071.stderr:207739: waiting osd.2 seq 279172874243 2021-07-26T06:42:14.153 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:42:14.153 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 279172874243 -lt 279172874243 2021-07-26T06:42:14.153 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:14.153 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-253403070471 2021-07-26T06:42:14.154 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:14.154 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:42:14.154 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-253403070471 2021-07-26T06:42:14.154 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:14.155 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=253403070471 2021-07-26T06:42:14.155 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 253403070471' 2021-07-26T06:42:14.155 INFO:tasks.workunit.client.0.smithi071.stderr:207739: waiting osd.3 seq 253403070471 2021-07-26T06:42:14.155 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:42:14.159 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 253403070472 -lt 253403070471 2021-07-26T06:42:14.159 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:42:14.159 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:14.159 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:14.160 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:42:14.160 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:42:14.160 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:42:14.160 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:42:14.161 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:42:14.161 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:42:14.161 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:42:14.161 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:42:14.161 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:42:14.162 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:42:14.162 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:14.162 INFO:tasks.workunit.client.0.smithi071.stderr:207739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:14.162 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:42:14.163 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:42:14.163 INFO:tasks.workunit.client.0.smithi071.stderr:207739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:42:14.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:42:14.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:42:14.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 207738 2021-07-26T06:42:14.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:42:14.164 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:42:14.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:42:14.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2021-07-26T06:42:14.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:42:14.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2021-07-26T06:42:14.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:42:14.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:42:14.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:42:14.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:42:14.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2021-07-26T06:42:14.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2021-07-26T06:42:14.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:42:14.419 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:42:14.419 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:42:14.419 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:42:14.419 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:42:14.420 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:42:14.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:41:42.144807+0000 2021-07-26T06:42:14.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:42:15.125 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:42:15.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:41:42.144807+0000 2021-07-26T06:42:15.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:42:15.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:41:42.144807+0000 2021-07-26T06:42:15.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:42:15.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:42:15.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:42:15.137 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:42:15.137 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:42:15.137 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:42:15.137 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:42:15.138 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:42:15.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:41:42.144807+0000 '>' 2021-07-26T06:41:42.144807+0000 2021-07-26T06:42:15.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:42:16.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:42:16.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:42:16.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:42:16.398 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:42:16.398 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:42:16.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:42:16.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:42:16.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:41:42.144807+0000 '>' 2021-07-26T06:41:42.144807+0000 2021-07-26T06:42:16.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:42:17.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:42:17.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:42:17.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:42:17.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:42:17.677 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:42:17.677 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:42:17.677 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:42:17.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:41:42.144807+0000 '>' 2021-07-26T06:41:42.144807+0000 2021-07-26T06:42:17.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:42:18.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:42:18.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:42:18.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:42:18.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:42:18.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:42:18.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:42:18.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:42:19.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:41:42.144807+0000 '>' 2021-07-26T06:41:42.144807+0000 2021-07-26T06:42:19.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:42:20.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:42:20.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:42:20.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:42:20.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:42:20.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:42:20.222 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:42:20.222 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:42:20.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:41:42.144807+0000 '>' 2021-07-26T06:41:42.144807+0000 2021-07-26T06:42:20.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:42:21.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:42:21.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:42:21.494 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:42:21.494 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:42:21.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:42:21.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:42:21.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:42:21.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:42:16.445757+0000 '>' 2021-07-26T06:41:42.144807+0000 2021-07-26T06:42:21.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:42:21.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2021-07-26T06:42:21.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:42:21.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:42:21.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:42:21.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 209290"' 2021-07-26T06:42:21.767 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 209290' 2021-07-26T06:42:21.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2021-07-26T06:42:21.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:42:21.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:42:21.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 209292"' 2021-07-26T06:42:21.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:42:21.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 209292' 2021-07-26T06:42:21.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2021-07-26T06:42:21.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 209290 209292' 2021-07-26T06:42:21.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:42:21.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:42:21.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/209293: /' 2021-07-26T06:42:21.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 209290 2021-07-26T06:42:21.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2021-07-26T06:42:21.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/209295: /' 2021-07-26T06:42:22.872 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:42:22.872 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:42:22.873 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:42:22.873 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:42:22.873 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:42:22.873 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:42:22.873 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:42:22.874 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:42:22.874 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:42:22.874 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:42:22.874 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:42:22.875 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:42:22.875 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:42:22.875 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:42:22.875 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:42:22.875 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:42:22.876 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:42:22.876 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:42:22.876 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:42:22.876 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:42:22.877 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:42:22.877 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:42:22.877 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2021-07-26T06:42:22.877 INFO:tasks.workunit.client.0.smithi071.stderr:209293: Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:42:22.877 INFO:tasks.workunit.client.0.smithi071.stderr:209293: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:42:22.878 INFO:tasks.workunit.client.0.smithi071.stderr:209293: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:42:22.878 INFO:tasks.workunit.client.0.smithi071.stderr:209293: Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:42:22.878 INFO:tasks.workunit.client.0.smithi071.stderr:209293: _ 2021-07-26T06:42:22.878 INFO:tasks.workunit.client.0.smithi071.stderr:209293: hinfo_key 2021-07-26T06:42:22.879 INFO:tasks.workunit.client.0.smithi071.stderr:209293: snapset 2021-07-26T06:42:22.880 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:42:22.880 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:42:22.881 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:42:22.881 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:42:22.881 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:42:22.881 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:42:22.882 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:42:22.882 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:42:22.882 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:42:22.883 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:42:22.883 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:42:22.883 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:42:22.883 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:42:22.884 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:42:22.884 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:42:22.884 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:42:22.884 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:42:22.884 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:42:22.885 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:42:22.885 INFO:tasks.workunit.client.0.smithi071.stderr:209293: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:42:22.885 INFO:tasks.workunit.client.0.smithi071.stderr:209293: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:22.885 INFO:tasks.workunit.client.0.smithi071.stderr:209293: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:22.886 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:22.886 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:22.886 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:42:22.886 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:42:22.886 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:42:22.887 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:42:22.887 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:42:22.887 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:22.887 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:42:22.888 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:42:22.888 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:42:22.888 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:42:22.888 INFO:tasks.workunit.client.0.smithi071.stderr:209293: start osd.1 2021-07-26T06:42:23.124 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:42:23.124 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:42:23.125 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=2 2021-07-26T06:42:23.125 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:42:23.125 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2021-07-26T06:42:23.125 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:42:23.125 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:42:23.126 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=2 2021-07-26T06:42:23.126 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:42:23.126 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2021-07-26T06:42:23.126 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:42:23.127 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:42:23.127 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:42:23.127 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:42:23.127 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:42:23.128 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:42:23.128 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2021-07-26T06:42:23.128 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:42:23.128 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:42:23.128 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=2 2021-07-26T06:42:23.129 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:42:23.129 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:42:23.129 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2021-07-26T06:42:23.129 INFO:tasks.workunit.client.0.smithi071.stderr:209295: Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:42:23.130 INFO:tasks.workunit.client.0.smithi071.stderr:209295: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:42:23.130 INFO:tasks.workunit.client.0.smithi071.stderr:209295: Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:42:23.130 INFO:tasks.workunit.client.0.smithi071.stderr:209295: Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T06:42:23.130 INFO:tasks.workunit.client.0.smithi071.stderr:209295: _ 2021-07-26T06:42:23.131 INFO:tasks.workunit.client.0.smithi071.stderr:209295: hinfo_key 2021-07-26T06:42:23.131 INFO:tasks.workunit.client.0.smithi071.stderr:209295: snapset 2021-07-26T06:42:23.131 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2021-07-26T06:42:23.131 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:42:23.131 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:42:23.132 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:42:23.132 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:42:23.132 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:42:23.133 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:42:23.133 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:42:23.133 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:42:23.134 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:42:23.134 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:42:23.134 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:42:23.135 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:42:23.135 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:42:23.135 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:42:23.135 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:42:23.136 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:42:23.136 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:42:23.136 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:42:23.136 INFO:tasks.workunit.client.0.smithi071.stderr:209295: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:42:23.136 INFO:tasks.workunit.client.0.smithi071.stderr:209295: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:23.137 INFO:tasks.workunit.client.0.smithi071.stderr:209295: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:23.138 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:23.138 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:23.138 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:42:23.139 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:42:23.139 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:42:23.140 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:42:23.140 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:42:23.140 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:23.140 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:42:23.141 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:42:23.141 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:42:23.141 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:42:23.142 INFO:tasks.workunit.client.0.smithi071.stderr:209295: start osd.2 2021-07-26T06:42:25.765 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:25.766 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/2/whoami 2021-07-26T06:42:25.766 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:42:25.766 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:42:25.767 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:42:25.767 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:42:25.767 INFO:tasks.workunit.client.0.smithi071.stderr:209295: 2021-07-26T06:42:23.149+0000 7f64f77fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:25.767 INFO:tasks.workunit.client.0.smithi071.stderr:209295: 2021-07-26T06:42:23.255+0000 7f64f77fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:25.768 INFO:tasks.workunit.client.0.smithi071.stderr:209295: 2021-07-26T06:42:23.256+0000 7f64f77fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:25.768 INFO:tasks.workunit.client.0.smithi071.stderr:209295: 2021-07-26T06:42:23.259+0000 7f64f77fc0c0 -1 Falling back to public interface 2021-07-26T06:42:25.768 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:42:25.769 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:42:25.769 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:42:25.769 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:42:25.770 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:42:25.770 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:25.772 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:42:25.772 INFO:tasks.workunit.client.0.smithi071.stderr:209295: 0 2021-07-26T06:42:25.772 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:25.773 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:42:25.773 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:42:25.773 INFO:tasks.workunit.client.0.smithi071.stderr:209295: 2021-07-26T06:42:24.017+0000 7f64f77fc0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:42:25.773 INFO:tasks.workunit.client.0.smithi071.stderr:209295: 2021-07-26T06:42:24.025+0000 7f64f77fc0c0 -1 journal do_read_entry(921600): bad header magic 2021-07-26T06:42:25.774 INFO:tasks.workunit.client.0.smithi071.stderr:209295: 2021-07-26T06:42:24.025+0000 7f64f77fc0c0 -1 journal do_read_entry(921600): bad header magic 2021-07-26T06:42:25.774 INFO:tasks.workunit.client.0.smithi071.stderr:209295: 2021-07-26T06:42:24.032+0000 7f64f77fc0c0 -1 osd.2 67 log_to_monitors {default=true} 2021-07-26T06:42:25.774 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:42:25.775 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:25.775 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:42:25.775 INFO:tasks.workunit.client.0.smithi071.stderr:209295: 1 2021-07-26T06:42:25.776 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:25.776 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:42:25.776 INFO:tasks.workunit.client.0.smithi071.stderr:209295: osd.2 up in weight 1 up_from 72 up_thru 49 down_at 68 last_clean_interval [65,67) [v2:127.0.0.1:6818/209487,v1:127.0.0.1:6819/209487] [v2:127.0.0.1:6820/209487,v1:127.0.0.1:6821/209487] exists,up 780522fb-cdae-4eb2-80e8-854a9ba99cbb 2021-07-26T06:42:25.777 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:42:25.777 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:42:25.777 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:42:25.778 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:25.778 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:42:25.778 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:42:25.778 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:42:25.779 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:42:25.779 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:42:25.779 INFO:tasks.workunit.client.0.smithi071.stderr:209293: 2021-07-26T06:42:22.899+0000 7fb4d8f9e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:25.780 INFO:tasks.workunit.client.0.smithi071.stderr:209293: 2021-07-26T06:42:22.902+0000 7fb4d8f9e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:25.780 INFO:tasks.workunit.client.0.smithi071.stderr:209293: 2021-07-26T06:42:22.903+0000 7fb4d8f9e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:25.781 INFO:tasks.workunit.client.0.smithi071.stderr:209293: 2021-07-26T06:42:22.904+0000 7fb4d8f9e0c0 -1 Falling back to public interface 2021-07-26T06:42:25.781 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:42:25.781 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:42:25.781 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:42:25.782 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:42:25.782 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:42:25.782 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:25.782 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:42:25.783 INFO:tasks.workunit.client.0.smithi071.stderr:209293: 0 2021-07-26T06:42:25.783 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:25.783 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:42:25.783 INFO:tasks.workunit.client.0.smithi071.stderr:209293: 2021-07-26T06:42:23.958+0000 7fb4d8f9e0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:42:25.783 INFO:tasks.workunit.client.0.smithi071.stderr:209293: 2021-07-26T06:42:23.967+0000 7fb4d8f9e0c0 -1 journal do_read_entry(999424): bad header magic 2021-07-26T06:42:25.784 INFO:tasks.workunit.client.0.smithi071.stderr:209293: 2021-07-26T06:42:23.967+0000 7fb4d8f9e0c0 -1 journal do_read_entry(999424): bad header magic 2021-07-26T06:42:25.784 INFO:tasks.workunit.client.0.smithi071.stderr:209293: 2021-07-26T06:42:23.974+0000 7fb4d8f9e0c0 -1 osd.1 67 log_to_monitors {default=true} 2021-07-26T06:42:25.784 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:42:25.784 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:42:25.785 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:25.785 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:42:25.785 INFO:tasks.workunit.client.0.smithi071.stderr:209293: 1 2021-07-26T06:42:25.785 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:25.785 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:42:25.786 INFO:tasks.workunit.client.0.smithi071.stderr:209293: osd.1 up in weight 1 up_from 72 up_thru 72 down_at 68 last_clean_interval [65,67) [v2:127.0.0.1:6810/209420,v1:127.0.0.1:6811/209420] [v2:127.0.0.1:6812/209420,v1:127.0.0.1:6813/209420] exists,up 31c57fb5-51ff-4cb0-a76a-87197652a01e 2021-07-26T06:42:25.786 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:42:25.786 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:42:25.786 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:42:26.997 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:42:26.997 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:42:26.997 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:42:26.998 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:42:26.998 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:42:26.998 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:42:26.998 INFO:tasks.workunit.client.0.smithi071.stderr:209293: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:42:26.999 INFO:tasks.workunit.client.0.smithi071.stderr:209293: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:42:26.999 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:42:26.999 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:42:26.999 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:42:27.000 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:42:27.000 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:42:27.000 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:42:27.000 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:42:27.001 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:42:27.001 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:42:27.001 INFO:tasks.workunit.client.0.smithi071.stderr:209293: 1 2021-07-26T06:42:27.002 INFO:tasks.workunit.client.0.smithi071.stderr:209293: 2 2021-07-26T06:42:27.002 INFO:tasks.workunit.client.0.smithi071.stderr:209293: 3' 2021-07-26T06:42:27.002 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:42:27.002 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:27.003 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:42:27.003 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836514 2021-07-26T06:42:27.003 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836514 2021-07-26T06:42:27.003 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836514' 2021-07-26T06:42:27.004 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:27.004 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:42:27.004 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=309237645314 2021-07-26T06:42:27.004 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 309237645314 2021-07-26T06:42:27.005 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836514 1-309237645314' 2021-07-26T06:42:27.005 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:27.005 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:42:27.006 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=309237645314 2021-07-26T06:42:27.006 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 309237645314 2021-07-26T06:42:27.006 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836514 1-309237645314 2-309237645314' 2021-07-26T06:42:27.006 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:27.007 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 2021-07-26T06:42:27.007 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:42:27.008 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:42:27.008 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:42:27.008 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:42:27.009 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:42:27.009 INFO:tasks.workunit.client.0.smithi071.stderr:209295: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:42:27.009 INFO:tasks.workunit.client.0.smithi071.stderr:209295: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:42:27.009 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:42:27.010 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:42:27.010 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:42:27.010 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:42:27.011 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:42:27.011 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:42:27.011 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:42:27.011 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:42:27.012 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:42:27.012 INFO:tasks.workunit.client.0.smithi071.stderr:209295: 1 2021-07-26T06:42:27.012 INFO:tasks.workunit.client.0.smithi071.stderr:209295: 2 2021-07-26T06:42:27.013 INFO:tasks.workunit.client.0.smithi071.stderr:209295: 3' 2021-07-26T06:42:27.013 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:42:27.013 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:27.013 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:42:27.013 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836515 2021-07-26T06:42:27.014 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836515 2021-07-26T06:42:27.014 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515' 2021-07-26T06:42:27.014 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:27.014 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:42:27.015 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=309237645315 2021-07-26T06:42:27.015 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 309237645315 2021-07-26T06:42:27.015 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-309237645315' 2021-07-26T06:42:27.015 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:27.016 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:42:27.016 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=309237645315 2021-07-26T06:42:27.016 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 309237645315 2021-07-26T06:42:27.016 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-309237645315 2-309237645315' 2021-07-26T06:42:27.016 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:28.995 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:42:28.996 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=253403070476 2021-07-26T06:42:28.997 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 253403070476 2021-07-26T06:42:28.997 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-309237645315 2-309237645315 3-253403070476' 2021-07-26T06:42:28.997 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:28.998 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836515 2021-07-26T06:42:28.998 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:28.998 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:42:28.999 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836515 2021-07-26T06:42:28.999 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:28.999 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836515 2021-07-26T06:42:29.000 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2021-07-26T06:42:29.000 INFO:tasks.workunit.client.0.smithi071.stderr:209295: waiting osd.0 seq 21474836515 2021-07-26T06:42:29.000 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:29.000 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836513 -lt 21474836515 2021-07-26T06:42:29.001 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:42:29.001 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:42:29.001 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:29.001 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836516 -lt 21474836515 2021-07-26T06:42:29.002 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:29.002 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-309237645315 2021-07-26T06:42:29.002 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:29.002 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:42:29.003 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-309237645315 2021-07-26T06:42:29.003 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:29.003 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=309237645315 2021-07-26T06:42:29.004 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 309237645315' 2021-07-26T06:42:29.004 INFO:tasks.workunit.client.0.smithi071.stderr:209295: waiting osd.1 seq 309237645315 2021-07-26T06:42:29.004 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:42:29.005 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 309237645315 -lt 309237645315 2021-07-26T06:42:29.005 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:29.005 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-309237645315 2021-07-26T06:42:29.006 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:29.006 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:42:29.006 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-309237645315 2021-07-26T06:42:29.007 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:29.007 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:flush_pg_stats 2021-07-26T06:42:29.008 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=253403070477 2021-07-26T06:42:29.008 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 253403070477 2021-07-26T06:42:29.008 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836514 1-309237645314 2-309237645314 3-253403070477' 2021-07-26T06:42:29.009 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:29.009 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836514 2021-07-26T06:42:29.009 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:29.010 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:42:29.010 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836514 2021-07-26T06:42:29.010 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:29.010 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836514 2021-07-26T06:42:29.011 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2021-07-26T06:42:29.011 INFO:tasks.workunit.client.0.smithi071.stderr:209293: waiting osd.0 seq 21474836514 2021-07-26T06:42:29.011 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:29.012 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836513 -lt 21474836514 2021-07-26T06:42:29.012 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:42:29.012 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:42:29.012 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:29.013 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836516 -lt 21474836514 2021-07-26T06:42:29.013 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:29.013 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-309237645314 2021-07-26T06:42:29.013 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:29.013 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:42:29.014 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-309237645314 2021-07-26T06:42:29.014 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:29.014 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=309237645314 2021-07-26T06:42:29.014 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 309237645314' 2021-07-26T06:42:29.015 INFO:tasks.workunit.client.0.smithi071.stderr:209293: waiting osd.1 seq 309237645314 2021-07-26T06:42:29.015 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:42:29.015 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 309237645315 -lt 309237645314 2021-07-26T06:42:29.015 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:29.016 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-309237645314 2021-07-26T06:42:29.016 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:29.016 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:42:29.016 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-309237645314 2021-07-26T06:42:29.017 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:31.001 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=309237645315 2021-07-26T06:42:31.002 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 309237645315' 2021-07-26T06:42:31.002 INFO:tasks.workunit.client.0.smithi071.stderr:209295: waiting osd.2 seq 309237645315 2021-07-26T06:42:31.002 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:42:31.002 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 309237645315 -lt 309237645315 2021-07-26T06:42:31.002 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:31.003 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-253403070476 2021-07-26T06:42:31.003 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:31.003 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:42:31.003 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-253403070476 2021-07-26T06:42:31.004 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:31.004 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=253403070476 2021-07-26T06:42:31.004 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 253403070476' 2021-07-26T06:42:31.004 INFO:tasks.workunit.client.0.smithi071.stderr:209295: waiting osd.3 seq 253403070476 2021-07-26T06:42:31.005 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:42:31.005 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 253403070477 -lt 253403070476 2021-07-26T06:42:31.005 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:42:31.005 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:31.006 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:31.006 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:42:31.006 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:42:31.006 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:42:31.006 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:42:31.007 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:42:31.007 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:42:31.007 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:42:31.007 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:42:31.008 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:42:31.008 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:42:31.008 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:31.008 INFO:tasks.workunit.client.0.smithi071.stderr:209295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:31.009 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:42:31.009 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:42:31.009 INFO:tasks.workunit.client.0.smithi071.stderr:209295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:42:31.009 INFO:tasks.workunit.client.0.smithi071.stderr:2086: flush_pg_stats: seq=309237645314 2021-07-26T06:42:31.009 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 309237645314' 2021-07-26T06:42:31.010 INFO:tasks.workunit.client.0.smithi071.stderr:209293: waiting osd.2 seq 309237645314 2021-07-26T06:42:31.010 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:42:31.010 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 309237645315 -lt 309237645314 2021-07-26T06:42:31.010 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:31.011 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-253403070477 2021-07-26T06:42:31.011 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:31.011 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:42:31.011 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-253403070477 2021-07-26T06:42:31.012 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:31.012 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=253403070477 2021-07-26T06:42:31.012 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 253403070477' 2021-07-26T06:42:31.012 INFO:tasks.workunit.client.0.smithi071.stderr:209293: waiting osd.3 seq 253403070477 2021-07-26T06:42:31.012 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:42:31.013 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 253403070477 -lt 253403070477 2021-07-26T06:42:31.013 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:42:31.013 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:31.013 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:31.014 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:42:31.014 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:42:31.014 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:42:31.014 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:42:31.014 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:42:31.015 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:42:31.015 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:42:31.015 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:42:31.015 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:42:31.016 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:42:31.016 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:31.016 INFO:tasks.workunit.client.0.smithi071.stderr:209293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:31.016 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:42:31.016 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:42:31.017 INFO:tasks.workunit.client.0.smithi071.stderr:209293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:42:31.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:42:31.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:42:31.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:42:31.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 209292 2021-07-26T06:42:31.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:42:31.018 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:42:31.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:42:31.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2021-07-26T06:42:31.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:42:31.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:42:31.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:42:31.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2021-07-26T06:42:31.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2021-07-26T06:42:31.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2021-07-26T06:42:31.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=3 2021-07-26T06:42:31.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=1 2021-07-26T06:42:31.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2021-07-26T06:42:31.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:42:31.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:42:31.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:42:31.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 210629"' 2021-07-26T06:42:31.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 210629' 2021-07-26T06:42:31.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:42:31.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:42:31.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:42:31.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 210631"' 2021-07-26T06:42:31.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:42:31.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 210631' 2021-07-26T06:42:31.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2021-07-26T06:42:31.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 210629 210631' 2021-07-26T06:42:31.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:42:31.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:42:31.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 210629 2021-07-26T06:42:31.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/210632: /' 2021-07-26T06:42:31.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:42:31.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/210634: /' 2021-07-26T06:42:32.602 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:42:32.603 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:42:32.603 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:42:32.603 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:42:32.604 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:42:32.604 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:42:32.604 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:42:32.604 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:42:32.605 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:42:32.605 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:42:32.605 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:42:32.605 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:42:32.606 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:42:32.606 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:42:32.606 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:42:32.606 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:42:32.606 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:42:32.607 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:42:32.607 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:42:32.608 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:42:32.608 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:42:32.609 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:42:32.609 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2021-07-26T06:42:32.609 INFO:tasks.workunit.client.0.smithi071.stderr:210632: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:42:32.609 INFO:tasks.workunit.client.0.smithi071.stderr:210632: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:42:32.610 INFO:tasks.workunit.client.0.smithi071.stderr:210632: Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:42:32.610 INFO:tasks.workunit.client.0.smithi071.stderr:210632: remove 0#2:eb822e21:::SOMETHING:head# 2021-07-26T06:42:32.610 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:42:32.610 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:42:32.611 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:42:32.611 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:42:32.611 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:42:32.611 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:42:32.611 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:42:32.612 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:42:32.612 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:42:32.612 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:42:32.612 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:42:32.613 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:42:32.613 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:42:32.613 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:42:32.614 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:42:32.614 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:42:32.614 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:42:32.614 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:42:32.615 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:42:32.615 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:42:32.615 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:42:32.615 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:42:32.616 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:42:32.616 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:42:32.616 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:42:32.617 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:42:32.617 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:42:32.617 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:42:32.617 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2021-07-26T06:42:32.618 INFO:tasks.workunit.client.0.smithi071.stderr:210634: Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:42:32.618 INFO:tasks.workunit.client.0.smithi071.stderr:210634: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:42:32.618 INFO:tasks.workunit.client.0.smithi071.stderr:210634: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:42:32.618 INFO:tasks.workunit.client.0.smithi071.stderr:210634: Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:42:32.619 INFO:tasks.workunit.client.0.smithi071.stderr:210634: remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:42:32.619 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:42:32.619 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:42:32.620 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:42:32.620 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:42:32.620 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:42:32.620 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:42:32.621 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:42:32.622 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:42:32.622 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:42:32.622 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:42:32.622 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:42:32.623 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:42:32.623 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:42:32.623 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:42:32.624 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:42:32.624 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:42:32.624 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:42:32.624 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:42:32.625 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:42:32.625 INFO:tasks.workunit.client.0.smithi071.stderr:210632: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:42:32.625 INFO:tasks.workunit.client.0.smithi071.stderr:210632: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:32.625 INFO:tasks.workunit.client.0.smithi071.stderr:210632: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:32.625 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:32.626 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:32.626 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:42:32.626 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:42:32.626 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:42:32.627 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:42:32.627 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:42:32.627 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:32.628 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:42:32.628 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:42:32.628 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:42:32.628 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:42:32.628 INFO:tasks.workunit.client.0.smithi071.stderr:210632: start osd.3 2021-07-26T06:42:32.629 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debargs=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:42:32.629 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:42:32.629 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:42:32.629 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:42:32.630 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:42:32.630 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:42:32.630 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:42:32.631 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:42:32.631 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:42:32.632 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:42:32.632 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:42:32.632 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:42:32.632 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:42:32.633 INFO:tasks.workunit.client.0.smithi071.stderr:210634: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:42:32.633 INFO:tasks.workunit.client.0.smithi071.stderr:210634: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:32.633 INFO:tasks.workunit.client.0.smithi071.stderr:210634: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:32.633 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:32.634 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:32.634 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:42:32.634 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:42:32.635 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:42:32.635 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:42:32.635 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:42:32.635 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:32.635 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:42:32.636 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:42:32.636 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:42:32.636 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:42:32.636 INFO:tasks.workunit.client.0.smithi071.stderr:210634: start osd.1 2021-07-26T06:42:34.924 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-diug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:34.925 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:42:34.925 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:42:34.925 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:42:34.926 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:42:34.926 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:42:34.926 INFO:tasks.workunit.client.0.smithi071.stderr:210632: 2021-07-26T06:42:32.633+0000 7fde4bde50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:34.926 INFO:tasks.workunit.client.0.smithi071.stderr:210632: 2021-07-26T06:42:32.636+0000 7fde4bde50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:34.927 INFO:tasks.workunit.client.0.smithi071.stderr:210632: 2021-07-26T06:42:32.637+0000 7fde4bde50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:34.927 INFO:tasks.workunit.client.0.smithi071.stderr:210632: 2021-07-26T06:42:32.638+0000 7fde4bde50c0 -1 Falling back to public interface 2021-07-26T06:42:34.927 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:42:34.927 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:42:34.928 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:42:34.928 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:42:34.928 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:42:34.928 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:34.928 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:42:34.929 INFO:tasks.workunit.client.0.smithi071.stderr:210632: 0 2021-07-26T06:42:34.929 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:34.929 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:42:34.929 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:42:34.930 INFO:tasks.workunit.client.0.smithi071.stderr:210632: 2021-07-26T06:42:33.405+0000 7fde4bde50c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:42:34.930 INFO:tasks.workunit.client.0.smithi071.stderr:210632: 2021-07-26T06:42:33.419+0000 7fde4bde50c0 -1 journal do_read_entry(1155072): bad header magic 2021-07-26T06:42:34.930 INFO:tasks.workunit.client.0.smithi071.stderr:210632: 2021-07-26T06:42:33.419+0000 7fde4bde50c0 -1 journal do_read_entry(1155072): bad header magic 2021-07-26T06:42:34.930 INFO:tasks.workunit.client.0.smithi071.stderr:210632: 2021-07-26T06:42:33.426+0000 7fde4bde50c0 -1 osd.3 74 log_to_monitors {default=true} 2021-07-26T06:42:34.931 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:42:34.931 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:34.931 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:42:34.931 INFO:tasks.workunit.client.0.smithi071.stderr:210632: 1 2021-07-26T06:42:34.931 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:34.932 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:42:34.932 INFO:tasks.workunit.client.0.smithi071.stderr:210632: osd.3 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [59,74) [v2:127.0.0.1:6811/210777,v1:127.0.0.1:6813/210777] [v2:127.0.0.1:6815/210777,v1:127.0.0.1:6817/210777] exists,up 8fb4ec75-d2c4-4729-b80a-1796fd4c97f9 2021-07-26T06:42:34.932 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:42:34.932 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:42:34.933 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:42:34.933 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:42:34.933 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_cler=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:34.933 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:42:34.934 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:42:34.934 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:42:34.934 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:42:34.934 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:42:34.935 INFO:tasks.workunit.client.0.smithi071.stderr:210634: 2021-07-26T06:42:32.632+0000 7f8a21ea30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:34.935 INFO:tasks.workunit.client.0.smithi071.stderr:210634: 2021-07-26T06:42:32.635+0000 7f8a21ea30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:34.935 INFO:tasks.workunit.client.0.smithi071.stderr:210634: 2021-07-26T06:42:32.636+0000 7f8a21ea30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:34.935 INFO:tasks.workunit.client.0.smithi071.stderr:210634: 2021-07-26T06:42:32.638+0000 7f8a21ea30c0 -1 Falling back to public interface 2021-07-26T06:42:34.936 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:42:34.936 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:42:34.936 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:42:34.936 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:42:34.936 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:42:34.937 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:34.937 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:42:34.937 INFO:tasks.workunit.client.0.smithi071.stderr:210634: 0 2021-07-26T06:42:34.937 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:34.938 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:42:34.938 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:42:34.938 INFO:tasks.workunit.client.0.smithi071.stderr:210634: 2021-07-26T06:42:33.374+0000 7f8a21ea30c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:42:34.938 INFO:tasks.workunit.client.0.smithi071.stderr:210634: 2021-07-26T06:42:33.385+0000 7f8a21ea30c0 -1 journal do_read_entry(1138688): bad header magic 2021-07-26T06:42:34.939 INFO:tasks.workunit.client.0.smithi071.stderr:210634: 2021-07-26T06:42:33.385+0000 7f8a21ea30c0 -1 journal do_read_entry(1138688): bad header magic 2021-07-26T06:42:34.939 INFO:tasks.workunit.client.0.smithi071.stderr:210634: 2021-07-26T06:42:33.392+0000 7f8a21ea30c0 -1 osd.1 74 log_to_monitors {default=true} 2021-07-26T06:42:34.939 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:42:34.939 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:34.940 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:42:34.940 INFO:tasks.workunit.client.0.smithi071.stderr:210634: 1 2021-07-26T06:42:34.940 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:34.940 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:42:34.940 INFO:tasks.workunit.client.0.smithi071.stderr:210634: osd.1 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [72,74) [v2:127.0.0.1:6810/210775,v1:127.0.0.1:6812/210775] [v2:127.0.0.1:6814/210775,v1:127.0.0.1:6816/210775] exists,up 31c57fb5-51ff-4cb0-a76a-87197652a01e 2021-07-26T06:42:34.941 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:42:34.941 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:42:34.941 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:42:34.941 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:42:35.722 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:42:35.722 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:42:35.722 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:42:35.723 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:42:35.723 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:42:35.723 INFO:tasks.workunit.client.0.smithi071.stderr:210634: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:42:35.723 INFO:tasks.workunit.client.0.smithi071.stderr:210634: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:42:35.723 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:42:35.724 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:42:35.724 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:42:35.724 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:42:35.724 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:42:35.725 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:42:35.725 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:42:35.725 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:42:35.725 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:42:35.726 INFO:tasks.workunit.client.0.smithi071.stderr:210634: 1 2021-07-26T06:42:35.726 INFO:tasks.workunit.client.0.smithi071.stderr:210634: 2 2021-07-26T06:42:35.726 INFO:tasks.workunit.client.0.smithi071.stderr:210634: 3' 2021-07-26T06:42:35.726 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:42:35.726 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:35.727 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:42:35.727 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836519 2021-07-26T06:42:35.727 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836519 2021-07-26T06:42:35.727 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519' 2021-07-26T06:42:35.727 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:35.728 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:42:35.728 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=335007449090 2021-07-26T06:42:35.728 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 335007449090 2021-07-26T06:42:35.728 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519 1-335007449090' 2021-07-26T06:42:35.728 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:35.729 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:42:35.729 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=309237645319 2021-07-26T06:42:35.729 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 309237645319 2021-07-26T06:42:35.729 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519 1-335007449090 2-309237645319' 2021-07-26T06:42:35.730 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:35.874 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pgan: local cmd= 2021-07-26T06:42:35.874 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:42:35.874 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:42:35.875 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:42:35.875 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:42:35.875 INFO:tasks.workunit.client.0.smithi071.stderr:210632: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:42:35.875 INFO:tasks.workunit.client.0.smithi071.stderr:210632: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:42:35.875 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:42:35.876 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:42:35.876 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:42:35.876 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:42:35.876 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:42:35.877 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:42:35.877 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:42:35.877 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:42:35.877 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:42:35.878 INFO:tasks.workunit.client.0.smithi071.stderr:210632: 1 2021-07-26T06:42:35.878 INFO:tasks.workunit.client.0.smithi071.stderr:210632: 2 2021-07-26T06:42:35.878 INFO:tasks.workunit.client.0.smithi071.stderr:210632: 3' 2021-07-26T06:42:35.878 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:42:35.878 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:35.879 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:42:35.879 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836518 2021-07-26T06:42:35.879 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836518 2021-07-26T06:42:35.879 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836518' 2021-07-26T06:42:35.879 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:35.880 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:42:35.880 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=335007449091 2021-07-26T06:42:35.880 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 335007449091 2021-07-26T06:42:35.880 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836518 1-335007449091' 2021-07-26T06:42:35.881 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:35.881 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:42:35.881 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=309237645318 2021-07-26T06:42:35.881 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 309237645318 2021-07-26T06:42:35.881 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836518 1-335007449091 2-309237645318' 2021-07-26T06:42:35.882 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:35.882 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:42:39.010 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=335007449090 2021-07-26T06:42:39.011 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 335007449090 2021-07-26T06:42:39.011 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836518 1-335007449091 2-309237645318 3-335007449090' 2021-07-26T06:42:39.011 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:39.012 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836518 2021-07-26T06:42:39.012 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:39.012 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:42:39.012 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836518 2021-07-26T06:42:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836518 2021-07-26T06:42:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2021-07-26T06:42:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:210632: waiting osd.0 seq 21474836518 2021-07-26T06:42:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:39.014 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836517 -lt 21474836518 2021-07-26T06:42:39.014 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:42:39.014 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:42:39.014 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:39.015 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836517 -lt 21474836518 2021-07-26T06:42:39.015 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:42:39.015 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:42:39.015 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:39.015 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836520 -lt 21474836518 2021-07-26T06:42:39.016 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:39.016 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-335007449091 2021-07-26T06:42:39.016 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:39.016 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:42:39.017 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-335007449091 2021-07-26T06:42:39.017 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:39.017 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=335007449091 2021-07-26T06:42:39.017 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 335007449091' 2021-07-26T06:42:39.017 INFO:tasks.workunit.client.0.smithi071.stderr:210632: waiting osd.1 seq 335007449091 2021-07-26T06:42:39.018 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:42:39.018 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 335007449091 -lt 335007449091 2021-07-26T06:42:39.018 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:39.018 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-309237645318 2021-07-26T06:42:39.019 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:39.056 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtes_stats 2021-07-26T06:42:39.057 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=335007449091 2021-07-26T06:42:39.057 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 335007449091 2021-07-26T06:42:39.057 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519 1-335007449090 2-309237645319 3-335007449091' 2021-07-26T06:42:39.057 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:39.058 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836519 2021-07-26T06:42:39.058 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:39.058 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:42:39.058 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836519 2021-07-26T06:42:39.059 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:39.059 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836519 2021-07-26T06:42:39.059 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2021-07-26T06:42:39.059 INFO:tasks.workunit.client.0.smithi071.stderr:210634: waiting osd.0 seq 21474836519 2021-07-26T06:42:39.059 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:39.060 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836517 -lt 21474836519 2021-07-26T06:42:39.060 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:42:39.060 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:42:39.060 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:39.061 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836517 -lt 21474836519 2021-07-26T06:42:39.061 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:42:39.061 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:42:39.061 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:39.062 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836520 -lt 21474836519 2021-07-26T06:42:39.062 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:39.062 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-335007449090 2021-07-26T06:42:39.062 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:39.063 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:42:39.064 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-335007449090 2021-07-26T06:42:39.064 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:39.064 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=335007449090 2021-07-26T06:42:39.064 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 335007449090' 2021-07-26T06:42:39.065 INFO:tasks.workunit.client.0.smithi071.stderr:210634: waiting osd.1 seq 335007449090 2021-07-26T06:42:39.065 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:42:39.065 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 335007449091 -lt 335007449090 2021-07-26T06:42:39.065 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:39.065 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-309237645319 2021-07-26T06:42:40.475 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:40.475 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:42:40.475 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-309237645319 2021-07-26T06:42:40.476 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:40.476 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=309237645319 2021-07-26T06:42:40.476 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 309237645319' 2021-07-26T06:42:40.476 INFO:tasks.workunit.client.0.smithi071.stderr:210634: waiting osd.2 seq 309237645319 2021-07-26T06:42:40.477 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:42:40.477 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 309237645319 -lt 309237645319 2021-07-26T06:42:40.477 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:40.477 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-335007449091 2021-07-26T06:42:40.478 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:40.478 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:42:40.478 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-335007449091 2021-07-26T06:42:40.478 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:40.478 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=335007449091 2021-07-26T06:42:40.479 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 335007449091' 2021-07-26T06:42:40.479 INFO:tasks.workunit.client.0.smithi071.stderr:210634: waiting osd.3 seq 335007449091 2021-07-26T06:42:40.479 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:42:40.480 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 335007449092 -lt 335007449091 2021-07-26T06:42:40.480 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:42:40.480 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:40.481 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:40.481 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:42:40.481 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:42:40.481 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:42:40.482 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:42:40.482 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:42:40.482 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:42:40.482 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:42:40.483 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:42:40.483 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:42:40.483 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:42:40.483 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:40.801 INFO:tasks.workunit.client.0.smithi071.stderr:210634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_t/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:42:40.801 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-309237645318 2021-07-26T06:42:40.802 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:40.802 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=309237645318 2021-07-26T06:42:40.803 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 309237645318' 2021-07-26T06:42:40.803 INFO:tasks.workunit.client.0.smithi071.stderr:210632: waiting osd.2 seq 309237645318 2021-07-26T06:42:40.803 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:42:40.803 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 309237645319 -lt 309237645318 2021-07-26T06:42:40.804 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:40.804 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-335007449090 2021-07-26T06:42:40.804 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:40.804 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:42:40.804 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-335007449090 2021-07-26T06:42:40.805 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:40.805 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=335007449090 2021-07-26T06:42:40.805 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 335007449090' 2021-07-26T06:42:40.805 INFO:tasks.workunit.client.0.smithi071.stderr:210632: waiting osd.3 seq 335007449090 2021-07-26T06:42:40.805 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:42:40.806 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 335007449092 -lt 335007449090 2021-07-26T06:42:40.806 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:42:40.806 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:40.806 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:40.807 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:42:40.807 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:42:40.807 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:42:40.807 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:42:40.807 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:42:40.808 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:42:40.808 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:42:40.808 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:42:40.808 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:42:40.809 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:42:40.809 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:40.809 INFO:tasks.workunit.client.0.smithi071.stderr:210632: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:40.809 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:42:40.809 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:42:40.810 INFO:tasks.workunit.client.0.smithi071.stderr:210632: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:42:40.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:42:40.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:42:40.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 210631 2021-07-26T06:42:40.813 INFO:tasks.workunit.client.0.smithi071.stderr:num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:40.813 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:42:40.814 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:42:40.814 INFO:tasks.workunit.client.0.smithi071.stderr:210634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:42:40.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:42:40.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:42:40.815 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:42:40.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:42:40.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2021-07-26T06:42:40.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:42:40.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2021-07-26T06:42:40.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:42:40.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:42:40.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:42:40.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:42:41.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2021-07-26T06:42:41.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2021-07-26T06:42:41.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:42:41.091 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:42:41.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:42:41.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:42:41.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:42:41.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:42:41.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:42:16.445757+0000 2021-07-26T06:42:41.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:42:41.606 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:42:41.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:42:16.445757+0000 2021-07-26T06:42:41.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:42:41.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:42:16.445757+0000 2021-07-26T06:42:41.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:42:41.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:42:41.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:42:41.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:42:41.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:42:41.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:42:41.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:42:41.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:42:41.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:42:16.445757+0000 '>' 2021-07-26T06:42:16.445757+0000 2021-07-26T06:42:41.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:42:42.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:42:42.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:42:42.881 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:42:42.881 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:42:42.881 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:42:42.881 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:42:42.882 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:42:43.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:42:16.445757+0000 '>' 2021-07-26T06:42:16.445757+0000 2021-07-26T06:42:43.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:42:44.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:42:44.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:42:44.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:42:44.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:42:44.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:42:44.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:42:44.156 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:42:44.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:42:16.445757+0000 '>' 2021-07-26T06:42:16.445757+0000 2021-07-26T06:42:44.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:42:45.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:42:45.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:42:45.428 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:42:45.428 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:42:45.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:42:45.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:42:45.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:42:45.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:42:43.333674+0000 '>' 2021-07-26T06:42:16.445757+0000 2021-07-26T06:42:45.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:42:45.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2021-07-26T06:42:45.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:42:45.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:42:45.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:42:45.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 212171"' 2021-07-26T06:42:45.710 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 212171' 2021-07-26T06:42:45.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:42:45.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:42:45.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:42:45.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 212173"' 2021-07-26T06:42:45.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:42:45.712 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 212173' 2021-07-26T06:42:45.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2021-07-26T06:42:45.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 212171 212173' 2021-07-26T06:42:45.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:42:45.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:42:45.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/212174: /' 2021-07-26T06:42:45.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 212171 2021-07-26T06:42:45.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:42:45.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/212176: /' 2021-07-26T06:42:47.459 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:42:47.459 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:42:47.460 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:42:47.460 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:42:47.460 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:42:47.460 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:42:47.461 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:42:47.461 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:42:47.461 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:42:47.461 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:42:47.462 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:42:47.462 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:42:47.462 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:42:47.462 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:42:47.463 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:42:47.463 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:42:47.463 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:42:47.463 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:42:47.464 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:42:47.464 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:42:47.464 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:42:47.464 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:42:47.464 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2021-07-26T06:42:47.465 INFO:tasks.workunit.client.0.smithi071.stderr:212176: Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:42:47.465 INFO:tasks.workunit.client.0.smithi071.stderr:212176: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:42:47.465 INFO:tasks.workunit.client.0.smithi071.stderr:212176: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:42:47.465 INFO:tasks.workunit.client.0.smithi071.stderr:212176: Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:42:47.466 INFO:tasks.workunit.client.0.smithi071.stderr:212176: _ 2021-07-26T06:42:47.466 INFO:tasks.workunit.client.0.smithi071.stderr:212176: hinfo_key 2021-07-26T06:42:47.466 INFO:tasks.workunit.client.0.smithi071.stderr:212176: snapset 2021-07-26T06:42:47.466 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:42:47.466 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:42:47.467 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:42:47.467 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:42:47.467 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:42:47.467 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:42:47.467 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: lo212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:42:47.468 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:42:47.468 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:42:47.468 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:42:47.468 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:42:47.469 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:42:47.469 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:42:47.469 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:42:47.469 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:42:47.469 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:42:47.470 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:42:47.470 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:42:47.470 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:42:47.470 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:42:47.470 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:42:47.471 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:42:47.471 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:42:47.471 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:42:47.471 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:42:47.472 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:42:47.472 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:42:47.472 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:42:47.472 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2021-07-26T06:42:47.472 INFO:tasks.workunit.client.0.smithi071.stderr:212174: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:42:47.473 INFO:tasks.workunit.client.0.smithi071.stderr:212174: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:42:47.473 INFO:tasks.workunit.client.0.smithi071.stderr:212174: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2021-07-26T06:42:47.473 INFO:tasks.workunit.client.0.smithi071.stderr:212174: Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:42:47.473 INFO:tasks.workunit.client.0.smithi071.stderr:212174: _ 2021-07-26T06:42:47.473 INFO:tasks.workunit.client.0.smithi071.stderr:212174: hinfo_key 2021-07-26T06:42:47.474 INFO:tasks.workunit.client.0.smithi071.stderr:212174: snapset 2021-07-26T06:42:47.474 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:42:47.474 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:42:47.474 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:42:47.475 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:42:47.475 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:42:47.475 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:42:47.476 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:42:47.476 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:42:47.476 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:42:47.477 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:42:47.477 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:42:47.477 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:42:47.477 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:42:47.478 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:42:47.478 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:42:47.478 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:42:47.478 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:42:47.478 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:42:47.479 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:42:47.479 INFO:tasks.workunit.client.0.smithi071.stderr:212174: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:42:47.479 INFO:tasks.workunit.client.0.smithi071.stderr:212174: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:47.479 INFO:tasks.workunit.client.0.smithi071.stderr:212174: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:47.479 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:47.480 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:47.480 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:42:47.480 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:42:47.480 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:42:47.481 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:42:47.481 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:42:47.481 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:47.481 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:42:47.481 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:42:47.482 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:42:47.482 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:42:47.482 INFO:tasks.workunit.client.0.smithi071.stderr:212174: start osd.3 2021-07-26T06:42:47.482 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-qucal 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:42:47.483 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:42:47.483 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:42:47.483 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:42:47.483 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:42:47.484 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:42:47.484 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:42:47.484 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:42:47.484 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:42:47.484 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:42:47.485 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:42:47.485 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:42:47.485 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:42:47.485 INFO:tasks.workunit.client.0.smithi071.stderr:212176: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:42:47.485 INFO:tasks.workunit.client.0.smithi071.stderr:212176: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:47.486 INFO:tasks.workunit.client.0.smithi071.stderr:212176: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:47.486 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:47.486 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:47.486 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:42:47.487 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:42:47.487 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:42:47.487 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:42:47.487 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:42:47.487 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:47.488 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:42:47.488 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:42:47.488 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:42:47.488 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:42:47.489 INFO:tasks.workunit.client.0.smithi071.stderr:212176: start osd.1 2021-07-26T06:42:49.528 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:49.528 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:42:49.528 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:42:49.529 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:42:49.529 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:42:49.529 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:42:49.529 INFO:tasks.workunit.client.0.smithi071.stderr:212176: 2021-07-26T06:42:47.487+0000 7fd7c6f610c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:49.530 INFO:tasks.workunit.client.0.smithi071.stderr:212176: 2021-07-26T06:42:47.489+0000 7fd7c6f610c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:49.530 INFO:tasks.workunit.client.0.smithi071.stderr:212176: 2021-07-26T06:42:47.490+0000 7fd7c6f610c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:49.530 INFO:tasks.workunit.client.0.smithi071.stderr:212176: 2021-07-26T06:42:47.492+0000 7fd7c6f610c0 -1 Falling back to public interface 2021-07-26T06:42:49.530 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:42:49.531 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:42:49.531 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:42:49.531 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:42:49.531 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:42:49.532 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:49.532 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:42:49.532 INFO:tasks.workunit.client.0.smithi071.stderr:212176: 0 2021-07-26T06:42:49.532 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:49.533 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:42:49.533 INFO:tasks.workunit.client.0.smithi071.stderr:212176: 2021-07-26T06:42:48.190+0000 7fd7c6f610c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:42:49.533 INFO:tasks.workunit.client.0.smithi071.stderr:212176: 2021-07-26T06:42:48.199+0000 7fd7c6f610c0 -1 journal do_read_entry(1290240): bad header magic 2021-07-26T06:42:49.533 INFO:tasks.workunit.client.0.smithi071.stderr:212176: 2021-07-26T06:42:48.199+0000 7fd7c6f610c0 -1 journal do_read_entry(1290240): bad header magic 2021-07-26T06:42:49.533 INFO:tasks.workunit.client.0.smithi071.stderr:212176: 2021-07-26T06:42:48.206+0000 7fd7c6f610c0 -1 osd.1 80 log_to_monitors {default=true} 2021-07-26T06:42:49.534 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:42:49.534 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:42:49.534 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:49.534 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:42:49.535 INFO:tasks.workunit.client.0.smithi071.stderr:212176: 1 2021-07-26T06:42:49.535 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:49.535 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:42:49.535 INFO:tasks.workunit.client.0.smithi071.stderr:212176: osd.1 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [78,80) [v2:127.0.0.1:6810/212318,v1:127.0.0.1:6812/212318] [v2:127.0.0.1:6814/212318,v1:127.0.0.1:6816/212318] exists,up 31c57fb5-51ff-4cb0-a76a-87197652a01e 2021-07-26T06:42:49.536 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:42:49.536 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:42:49.536 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:42:49.537 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_cleeue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:49.537 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:42:49.537 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:42:49.537 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:42:49.538 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:42:49.538 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:42:49.538 INFO:tasks.workunit.client.0.smithi071.stderr:212174: 2021-07-26T06:42:47.487+0000 7f763c0260c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:49.538 INFO:tasks.workunit.client.0.smithi071.stderr:212174: 2021-07-26T06:42:47.489+0000 7f763c0260c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:49.538 INFO:tasks.workunit.client.0.smithi071.stderr:212174: 2021-07-26T06:42:47.490+0000 7f763c0260c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:49.539 INFO:tasks.workunit.client.0.smithi071.stderr:212174: 2021-07-26T06:42:47.492+0000 7f763c0260c0 -1 Falling back to public interface 2021-07-26T06:42:49.539 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:42:49.539 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:42:49.540 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:42:49.540 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:42:49.540 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:42:49.541 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:49.541 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:42:49.542 INFO:tasks.workunit.client.0.smithi071.stderr:212174: 0 2021-07-26T06:42:49.542 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:49.542 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:42:49.543 INFO:tasks.workunit.client.0.smithi071.stderr:212174: 2021-07-26T06:42:48.190+0000 7f763c0260c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:42:49.543 INFO:tasks.workunit.client.0.smithi071.stderr:212174: 2021-07-26T06:42:48.199+0000 7f763c0260c0 -1 journal do_read_entry(1277952): bad header magic 2021-07-26T06:42:49.543 INFO:tasks.workunit.client.0.smithi071.stderr:212174: 2021-07-26T06:42:48.199+0000 7f763c0260c0 -1 journal do_read_entry(1277952): bad header magic 2021-07-26T06:42:49.543 INFO:tasks.workunit.client.0.smithi071.stderr:212174: 2021-07-26T06:42:48.205+0000 7f763c0260c0 -1 osd.3 80 log_to_monitors {default=true} 2021-07-26T06:42:49.544 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:42:49.544 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:42:49.545 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:42:49.545 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:42:49.545 INFO:tasks.workunit.client.0.smithi071.stderr:212174: 1 2021-07-26T06:42:49.545 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:42:49.546 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:42:49.546 INFO:tasks.workunit.client.0.smithi071.stderr:212174: osd.3 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [78,80) [v2:127.0.0.1:6811/212317,v1:127.0.0.1:6813/212317] [v2:127.0.0.1:6815/212317,v1:127.0.0.1:6817/212317] exists,up 8fb4ec75-d2c4-4729-b80a-1796fd4c97f9 2021-07-26T06:42:49.546 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:42:49.546 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:42:49.547 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:42:50.300 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:42:50.300 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:42:50.300 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:42:50.301 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:42:50.301 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:42:50.301 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:42:50.301 INFO:tasks.workunit.client.0.smithi071.stderr:212174: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:42:50.302 INFO:tasks.workunit.client.0.smithi071.stderr:212174: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:42:50.302 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:42:50.302 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:42:50.302 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:42:50.303 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:42:50.303 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:42:50.303 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:42:50.303 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:42:50.304 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:42:50.304 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:42:50.304 INFO:tasks.workunit.client.0.smithi071.stderr:212174: 1 2021-07-26T06:42:50.304 INFO:tasks.workunit.client.0.smithi071.stderr:212174: 2 2021-07-26T06:42:50.305 INFO:tasks.workunit.client.0.smithi071.stderr:212174: 3' 2021-07-26T06:42:50.305 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:42:50.305 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:50.305 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:42:50.305 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836523 2021-07-26T06:42:50.306 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836523 2021-07-26T06:42:50.306 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836523' 2021-07-26T06:42:50.306 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:50.306 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:42:50.307 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=360777252866 2021-07-26T06:42:50.307 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 360777252866 2021-07-26T06:42:50.307 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836523 1-360777252866' 2021-07-26T06:42:50.307 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:50.308 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:42:50.308 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=309237645324 2021-07-26T06:42:50.308 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 309237645324 2021-07-26T06:42:50.308 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836523 1-360777252866 2-309237645324' 2021-07-26T06:42:50.309 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:50.309 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: an 2021-07-26T06:42:50.309 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:42:50.310 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:42:50.310 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:42:50.310 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:42:50.311 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:42:50.311 INFO:tasks.workunit.client.0.smithi071.stderr:212176: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:42:50.311 INFO:tasks.workunit.client.0.smithi071.stderr:212176: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:42:50.311 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:42:50.312 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:42:50.312 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:42:50.312 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:42:50.312 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:42:50.312 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:42:50.313 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:42:50.313 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:42:50.313 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:42:50.313 INFO:tasks.workunit.client.0.smithi071.stderr:212176: 1 2021-07-26T06:42:50.314 INFO:tasks.workunit.client.0.smithi071.stderr:212176: 2 2021-07-26T06:42:50.314 INFO:tasks.workunit.client.0.smithi071.stderr:212176: 3' 2021-07-26T06:42:50.314 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:42:50.314 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:50.315 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:42:50.315 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836524 2021-07-26T06:42:50.315 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836524 2021-07-26T06:42:50.315 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524' 2021-07-26T06:42:50.316 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:50.316 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:42:50.316 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=360777252867 2021-07-26T06:42:50.316 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 360777252867 2021-07-26T06:42:50.316 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-360777252867' 2021-07-26T06:42:50.317 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:50.317 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:42:50.317 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=309237645323 2021-07-26T06:42:50.317 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 309237645323 2021-07-26T06:42:50.318 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-360777252867 2-309237645323' 2021-07-26T06:42:50.318 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:42:52.346 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd. ceph tell osd.3 flush_pg_stats 2021-07-26T06:42:52.346 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=360777252866 2021-07-26T06:42:52.347 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 360777252866 2021-07-26T06:42:52.347 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836523 1-360777252866 2-309237645324 3-360777252866' 2021-07-26T06:42:52.347 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:52.348 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836523 2021-07-26T06:42:52.348 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:52.348 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:42:52.348 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836523 2021-07-26T06:42:52.349 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:52.349 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836523 2021-07-26T06:42:52.349 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2021-07-26T06:42:52.350 INFO:tasks.workunit.client.0.smithi071.stderr:212174: waiting osd.0 seq 21474836523 2021-07-26T06:42:52.350 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:52.350 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836522 -lt 21474836523 2021-07-26T06:42:52.350 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:42:52.351 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:42:52.351 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:52.351 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836524 -lt 21474836523 2021-07-26T06:42:52.351 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:52.352 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-360777252866 2021-07-26T06:42:52.352 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:52.352 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:42:52.352 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-360777252866 2021-07-26T06:42:52.352 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:52.353 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=360777252866 2021-07-26T06:42:52.353 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 360777252866' 2021-07-26T06:42:52.353 INFO:tasks.workunit.client.0.smithi071.stderr:212174: waiting osd.1 seq 360777252866 2021-07-26T06:42:52.353 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:42:52.353 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 360777252867 -lt 360777252866 2021-07-26T06:42:52.354 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:52.354 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-309237645324 2021-07-26T06:42:52.354 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:52.354 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:42:52.355 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-309237645324 2021-07-26T06:42:52.355 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:52.355 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone3 flush_pg_stats 2021-07-26T06:42:52.355 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=360777252867 2021-07-26T06:42:52.355 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 360777252867 2021-07-26T06:42:52.356 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-360777252867 2-309237645323 3-360777252867' 2021-07-26T06:42:52.356 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:52.356 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836524 2021-07-26T06:42:52.356 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:52.357 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:42:52.357 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836524 2021-07-26T06:42:52.357 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:52.357 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836524 2021-07-26T06:42:52.357 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2021-07-26T06:42:52.358 INFO:tasks.workunit.client.0.smithi071.stderr:212176: waiting osd.0 seq 21474836524 2021-07-26T06:42:52.358 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:52.358 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836522 -lt 21474836524 2021-07-26T06:42:52.358 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:42:52.359 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:42:52.359 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:42:52.359 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836524 -lt 21474836524 2021-07-26T06:42:52.360 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:52.360 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-360777252867 2021-07-26T06:42:52.360 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:52.360 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:42:52.360 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-360777252867 2021-07-26T06:42:52.361 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:52.361 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=360777252867 2021-07-26T06:42:52.361 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 360777252867' 2021-07-26T06:42:52.361 INFO:tasks.workunit.client.0.smithi071.stderr:212176: waiting osd.1 seq 360777252867 2021-07-26T06:42:52.362 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:42:52.362 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 360777252867 -lt 360777252867 2021-07-26T06:42:52.362 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:52.362 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-309237645323 2021-07-26T06:42:52.362 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:52.363 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:42:52.363 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-309237645323 2021-07-26T06:42:52.363 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:53.948 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=309237645323 2021-07-26T06:42:53.948 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 309237645323' 2021-07-26T06:42:53.949 INFO:tasks.workunit.client.0.smithi071.stderr:212176: waiting osd.2 seq 309237645323 2021-07-26T06:42:53.949 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:42:53.949 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 309237645324 -lt 309237645323 2021-07-26T06:42:53.949 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:53.949 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-360777252867 2021-07-26T06:42:53.950 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:53.950 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:42:53.950 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-360777252867 2021-07-26T06:42:53.950 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:53.951 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=360777252867 2021-07-26T06:42:53.951 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 360777252867' 2021-07-26T06:42:53.951 INFO:tasks.workunit.client.0.smithi071.stderr:212176: waiting osd.3 seq 360777252867 2021-07-26T06:42:53.951 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:42:53.952 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 360777252867 -lt 360777252867 2021-07-26T06:42:53.952 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:42:53.952 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:53.952 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:53.952 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:42:53.953 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:42:53.953 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:42:53.953 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:42:53.954 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:42:53.954 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:42:53.954 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:42:53.954 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:42:53.955 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:42:53.955 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:42:53.955 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:53.955 INFO:tasks.workunit.client.0.smithi071.stderr:212176: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:53.955 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:42:53.956 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:42:53.956 INFO:tasks.workunit.client.0.smithi071.stderr:212176: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:42:53.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:42:54.048 INFO:tasks.workunit.client.0.smithi071.stderr:/ceph-helpers.sh:2086: flush_pg_stats: seq=309237645324 2021-07-26T06:42:54.048 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 309237645324' 2021-07-26T06:42:54.048 INFO:tasks.workunit.client.0.smithi071.stderr:212174: waiting osd.2 seq 309237645324 2021-07-26T06:42:54.049 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:42:54.049 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 309237645324 -lt 309237645324 2021-07-26T06:42:54.049 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:42:54.049 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-360777252866 2021-07-26T06:42:54.050 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:42:54.050 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:42:54.050 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-360777252866 2021-07-26T06:42:54.050 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:42:54.051 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=360777252866 2021-07-26T06:42:54.051 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 360777252866' 2021-07-26T06:42:54.051 INFO:tasks.workunit.client.0.smithi071.stderr:212174: waiting osd.3 seq 360777252866 2021-07-26T06:42:54.052 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:42:54.052 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 360777252867 -lt 360777252866 2021-07-26T06:42:54.052 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:42:54.052 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:54.053 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:54.053 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:42:54.053 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:42:54.053 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:42:54.054 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:42:54.054 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:42:54.054 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:42:54.055 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:42:54.055 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:42:54.055 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:42:54.056 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:42:54.056 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:42:54.056 INFO:tasks.workunit.client.0.smithi071.stderr:212174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:42:54.056 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:42:54.057 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:42:54.057 INFO:tasks.workunit.client.0.smithi071.stderr:212174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:42:54.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:42:54.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:42:54.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 212173 2021-07-26T06:42:54.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:42:54.058 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:42:54.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:42:54.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2021-07-26T06:42:54.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:42:54.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:42:54.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:42:54.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:42:54.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:42:54.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:42:54.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:42:54.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:42:54.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:42:54.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:42:54.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:42:54.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:42:54.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:42:54.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:42:54.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:42:54.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:42:54.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:42:54.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:42:54.206 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:42:54.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:42:54.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:42:54.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:42:54.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:42:54.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:42:54.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:42:54.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:42:54.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:42:54.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:54.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:54.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:42:54.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:42:54.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:42:54.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:42:54.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:42:54.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:42:54.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:42:54.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:42:54.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:42:54.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:42:54.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:42:54.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:42:54.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:42:54.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:42:54.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:42:54.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:42:54.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:42:54.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:42:54.243 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:42:54.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:42:54.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:42:54.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:42:54.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:42:54.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:42:54.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:42:54.245 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:42:54.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:42:54.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:42:54.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:42:54.247 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:42:54.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:54.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:54.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:42:54.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:42:54.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:42:54.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:42:54.249 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:42:54.249 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:54.250 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:54.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:42:54.252 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:42:54.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:42:54.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:42:54.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:42:54.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2021-07-26T06:42:54.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2021-07-26T06:42:54.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:821: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2021-07-26T06:42:54.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:795: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2021-07-26T06:42:54.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: corrupt_and_repair_jerasure: local allow_overwrites=true 2021-07-26T06:42:54.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:797: corrupt_and_repair_jerasure: local poolname=ecpool 2021-07-26T06:42:54.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2021-07-26T06:42:54.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:42:54.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:42:54.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:42:54.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:42:54.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:42:54.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T06:42:54.668 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:42:54.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:42:54.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:42:54.669 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:42:54.669 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:54.669 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:54.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:54.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:42:54.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:42:54.987 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:42:54.987 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:42:54.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:42:54.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:42:54.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:42:54.988 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:42:54.989 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:42:54.989 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:42:54.990 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:42:54.990 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:54.990 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:54.991 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:42:54.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:42:54.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:42:55.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:42:55.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:42:55.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:42:55.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:42:55.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:42:55.087 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:42:55.087 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:42:55.087 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:42:55.087 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:42:55.088 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:55.088 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:55.088 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:42:55.088 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:42:55.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:42:55.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2021-07-26T06:42:55.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:42:55.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:42:55.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:42:55.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:42:55.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:42:55.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:42:56.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:42:56.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:42:56.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:42:56.040 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:42:56.040 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:56.041 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:56.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:56.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:42:56.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:42:56.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: corrupt_and_repair_jerasure: seq 0 3 2021-07-26T06:42:56.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2021-07-26T06:42:56.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: '[' true = true ']' 2021-07-26T06:42:56.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2021-07-26T06:42:56.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:42:56.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:42:56.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:42:56.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:42:56.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:42:56.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:42:56.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:42:56.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:42:56.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:42:56.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:42:56.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:42:56.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:42:56.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:42:56.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:42:56.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:42:56.073 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:42:56.074 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:42:56.074 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:42:56.074 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:42:56.074 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:42:56.075 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:42:56.075 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:56.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:42:56.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:42:56.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:42:56.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:42:56.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:42:56.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:42:56.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:42:56.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:42:56.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:42:56.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:42:56.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:42:56.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:42:56.078 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:42:56.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d7a43020-3627-4729-a65f-0dd85d07fd1d 2021-07-26T06:42:56.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d7a43020-3627-4729-a65f-0dd85d07fd1d' 2021-07-26T06:42:56.079 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 d7a43020-3627-4729-a65f-0dd85d07fd1d 2021-07-26T06:42:56.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:42:56.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwWf5guUgEBRAA6qLORfEgvnBQzwEczae4yw== 2021-07-26T06:42:56.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwWf5guUgEBRAA6qLORfEgvnBQzwEczae4yw=="}' 2021-07-26T06:42:56.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d7a43020-3627-4729-a65f-0dd85d07fd1d -i td/osd-scrub-repair/0/new.json 2021-07-26T06:42:56.428 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:42:56.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:42:56.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBwWf5guUgEBRAA6qLORfEgvnBQzwEczae4yw== --osd-uuid d7a43020-3627-4729-a65f-0dd85d07fd1d 2021-07-26T06:42:56.463 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:42:56.464+0000 7ff78e6a80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:56.465 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:42:56.465+0000 7ff78e6a80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:56.466 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:42:56.466+0000 7ff78e6a80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:42:56.467 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:42:56.467+0000 7ff78e6a80c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:42:56.467 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:42:56.467+0000 7ff78e6a80c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:42:56.467 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:42:56.467+0000 7ff78e6a80c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:42:56.528 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:42:56.528+0000 7ff78e6a80c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T06:43:02.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:43:02.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:43:02.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:43:02.453 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:43:02.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:43:02.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:43:02.796 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:43:02.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:43:02.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:43:02.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:43:02.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:43:02.818 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:02.819+0000 7f27964cc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:02.853 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:02.854+0000 7f27964cc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:02.853 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:02.855+0000 7f27964cc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:03.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:43:03.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:43:03.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:43:03.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:43:03.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:43:03.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:03.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:43:03.069 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:43:03.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:03.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:43:03.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:43:03.381 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:03.383+0000 7f27964cc0c0 -1 Falling back to public interface 2021-07-26T06:43:04.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:43:04.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:04.364 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:43:04.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:43:04.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:04.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:43:04.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:43:05.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:43:05.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:05.644 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:43:05.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:43:05.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:05.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:43:05.753 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:05.754+0000 7f27964cc0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:43:05.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:43:06.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:43:06.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:06.970 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:43:06.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:43:06.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:06.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:43:07.290 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/214662,v1:127.0.0.1:6803/214662] [v2:127.0.0.1:6804/214662,v1:127.0.0.1:6805/214662] exists,up d7a43020-3627-4729-a65f-0dd85d07fd1d 2021-07-26T06:43:07.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:43:07.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:43:07.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:43:07.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2021-07-26T06:43:07.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: '[' true = true ']' 2021-07-26T06:43:07.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2021-07-26T06:43:07.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:43:07.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:43:07.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:43:07.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:43:07.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:43:07.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:43:07.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:43:07.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:43:07.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:43:07.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:43:07.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:43:07.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:43:07.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:43:07.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:43:07.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:43:07.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:43:07.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:43:07.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:43:07.313 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:43:07.314 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:43:07.314 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:43:07.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:43:07.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:43:07.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:43:07.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:43:07.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:43:07.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:43:07.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:43:07.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:43:07.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:43:07.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:43:07.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:43:07.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:43:07.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:43:07.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:43:07.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6a7f7fc8-4b60-487d-84b4-acde9f69054f 2021-07-26T06:43:07.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6a7f7fc8-4b60-487d-84b4-acde9f69054f' 2021-07-26T06:43:07.322 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 6a7f7fc8-4b60-487d-84b4-acde9f69054f 2021-07-26T06:43:07.324 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:43:07.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB7Wf5ggGxuEhAAAcL+vZB4A23GywjudbKYdg== 2021-07-26T06:43:07.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB7Wf5ggGxuEhAAAcL+vZB4A23GywjudbKYdg=="}' 2021-07-26T06:43:07.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6a7f7fc8-4b60-487d-84b4-acde9f69054f -i td/osd-scrub-repair/1/new.json 2021-07-26T06:43:07.731 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:43:07.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:43:07.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB7Wf5ggGxuEhAAAcL+vZB4A23GywjudbKYdg== --osd-uuid 6a7f7fc8-4b60-487d-84b4-acde9f69054f 2021-07-26T06:43:07.768 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:07.769+0000 7f96ad8e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:07.799 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:07.801+0000 7f96ad8e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:07.800 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:07.801+0000 7f96ad8e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:07.801 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:07.802+0000 7f96ad8e60c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:43:07.801 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:07.802+0000 7f96ad8e60c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:43:07.802 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:07.802+0000 7f96ad8e60c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:43:07.883 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:07.884+0000 7f96ad8e60c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T06:43:14.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:43:14.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:43:14.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:43:14.326 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:43:14.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:43:14.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:43:14.714 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:43:14.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:43:14.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:43:14.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:43:14.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:43:14.735 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:14.737+0000 7fe8a231b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:14.737 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:14.738+0000 7fe8a231b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:14.738 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:14.739+0000 7fe8a231b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:15.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:43:15.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:43:15.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:43:15.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:43:15.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:43:15.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:15.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:43:15.025 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:43:15.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:15.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:43:15.257 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:15.258+0000 7fe8a231b0c0 -1 Falling back to public interface 2021-07-26T06:43:15.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:43:16.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:43:16.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:16.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:43:16.370 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:43:16.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:16.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:43:16.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:43:17.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:43:17.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:17.646 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:43:17.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:43:17.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:17.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:43:17.772 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:17.773+0000 7fe8a231b0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:43:17.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:43:18.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:43:18.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:18.981 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:43:18.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:43:18.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:18.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:43:19.250 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/216061,v1:127.0.0.1:6811/216061] [v2:127.0.0.1:6812/216061,v1:127.0.0.1:6813/216061] exists,up 6a7f7fc8-4b60-487d-84b4-acde9f69054f 2021-07-26T06:43:19.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:43:19.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:43:19.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:43:19.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2021-07-26T06:43:19.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: '[' true = true ']' 2021-07-26T06:43:19.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2021-07-26T06:43:19.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:43:19.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:43:19.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:43:19.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:43:19.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:43:19.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:43:19.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:43:19.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:43:19.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:43:19.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:43:19.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:43:19.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:43:19.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:43:19.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:43:19.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:43:19.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:43:19.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:43:19.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:43:19.264 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:43:19.264 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:43:19.265 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:43:19.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:43:19.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:43:19.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:43:19.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:43:19.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:43:19.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:43:19.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:43:19.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:43:19.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:43:19.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:43:19.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:43:19.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:43:19.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:43:19.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:43:19.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a1dc4974-84bf-431c-9de1-8a3eb6ab83ae 2021-07-26T06:43:19.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a1dc4974-84bf-431c-9de1-8a3eb6ab83ae' 2021-07-26T06:43:19.271 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 a1dc4974-84bf-431c-9de1-8a3eb6ab83ae 2021-07-26T06:43:19.272 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:43:19.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHWf5grNQXEBAA3evtc8bGAjnaQ3WcwfW1ZA== 2021-07-26T06:43:19.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHWf5grNQXEBAA3evtc8bGAjnaQ3WcwfW1ZA=="}' 2021-07-26T06:43:19.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a1dc4974-84bf-431c-9de1-8a3eb6ab83ae -i td/osd-scrub-repair/2/new.json 2021-07-26T06:43:19.701 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:43:19.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2021-07-26T06:43:19.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCHWf5grNQXEBAA3evtc8bGAjnaQ3WcwfW1ZA== --osd-uuid a1dc4974-84bf-431c-9de1-8a3eb6ab83ae 2021-07-26T06:43:19.737 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:19.738+0000 7fb1eefe50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:19.760 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:19.762+0000 7fb1eefe50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:19.761 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:19.762+0000 7fb1eefe50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:19.762 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:19.763+0000 7fb1eefe50c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:43:19.762 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:19.763+0000 7fb1eefe50c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:43:19.763 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:19.763+0000 7fb1eefe50c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:43:19.834 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:19.835+0000 7fb1eefe50c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2021-07-26T06:43:26.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T06:43:26.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:43:26.042 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:43:26.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:43:26.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:43:26.379 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:43:26.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:43:26.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:43:26.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:43:26.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:43:26.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:43:26.401 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:26.402+0000 7f6facfd60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:26.403 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:26.404+0000 7f6facfd60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:26.404 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:26.405+0000 7f6facfd60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:26.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:43:26.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:43:26.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:43:26.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:43:26.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:43:26.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:26.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:43:26.654 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:43:26.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:26.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:43:26.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:43:26.933 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:26.934+0000 7f6facfd60c0 -1 Falling back to public interface 2021-07-26T06:43:27.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:43:27.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:27.919 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:43:27.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:43:27.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:27.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:43:28.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:43:29.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:43:29.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:29.197 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:43:29.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:43:29.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:29.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:43:29.417 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:29.418+0000 7f6facfd60c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:43:29.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:43:30.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:43:30.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:30.554 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:43:30.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:43:30.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:30.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:43:30.871 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/217468,v1:127.0.0.1:6819/217468] [v2:127.0.0.1:6820/217468,v1:127.0.0.1:6821/217468] exists,up a1dc4974-84bf-431c-9de1-8a3eb6ab83ae 2021-07-26T06:43:30.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:43:30.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:43:30.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:43:30.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2021-07-26T06:43:30.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: '[' true = true ']' 2021-07-26T06:43:30.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2021-07-26T06:43:30.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:43:30.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:43:30.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T06:43:30.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:43:30.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:43:30.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:43:30.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:43:30.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:43:30.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:43:30.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:43:30.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:43:30.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:43:30.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:43:30.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:43:30.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:43:30.882 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:43:30.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:43:30.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:43:30.884 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:43:30.884 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:43:30.884 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:43:30.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:43:30.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:43:30.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:43:30.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:43:30.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:43:30.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:43:30.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:43:30.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:43:30.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:43:30.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:43:30.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:43:30.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:43:30.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:43:30.888 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:43:30.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b8fbe2eb-a90e-4112-b4de-1f8c84490d2b 2021-07-26T06:43:30.888 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 b8fbe2eb-a90e-4112-b4de-1f8c84490d2b 2021-07-26T06:43:30.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b8fbe2eb-a90e-4112-b4de-1f8c84490d2b' 2021-07-26T06:43:30.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:43:30.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCSWf5g0CMYNRAAWY/B6rfTG1CYoDF6pXzVzA== 2021-07-26T06:43:30.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCSWf5g0CMYNRAAWY/B6rfTG1CYoDF6pXzVzA=="}' 2021-07-26T06:43:30.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b8fbe2eb-a90e-4112-b4de-1f8c84490d2b -i td/osd-scrub-repair/3/new.json 2021-07-26T06:43:31.321 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:43:31.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2021-07-26T06:43:31.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCSWf5g0CMYNRAAWY/B6rfTG1CYoDF6pXzVzA== --osd-uuid b8fbe2eb-a90e-4112-b4de-1f8c84490d2b 2021-07-26T06:43:31.357 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:31.358+0000 7fc2c54880c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:31.379 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:31.380+0000 7fc2c54880c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:31.380 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:31.381+0000 7fc2c54880c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:31.380 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:31.381+0000 7fc2c54880c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2021-07-26T06:43:31.381 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:31.381+0000 7fc2c54880c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2021-07-26T06:43:31.381 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:31.381+0000 7fc2c54880c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2021-07-26T06:43:31.452 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:31.453+0000 7fc2c54880c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2021-07-26T06:43:37.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2021-07-26T06:43:37.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:43:37.849 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd3 key to auth repository 2021-07-26T06:43:37.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T06:43:37.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:43:38.187 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.3 2021-07-26T06:43:38.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T06:43:38.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:43:38.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:43:38.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:43:38.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:43:38.210 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:38.210+0000 7fac7b6e80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:38.211 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:38.211+0000 7fac7b6e80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:38.212 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:38.212+0000 7fac7b6e80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:43:38.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T06:43:38.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:43:38.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:43:38.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:43:38.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:43:38.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:38.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:43:38.462 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:43:38.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:38.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:43:38.730 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:38.730+0000 7fac7b6e80c0 -1 Falling back to public interface 2021-07-26T06:43:38.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:43:39.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:43:39.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:39.817 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:43:39.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:43:39.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:39.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:43:40.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:43:41.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:43:41.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:41.095 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:43:41.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:43:41.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:41.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:43:41.198 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:43:41.198+0000 7fac7b6e80c0 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T06:43:41.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:43:42.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:43:42.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:43:42.516 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:43:42.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:43:42.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:43:42.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:43:42.790 INFO:tasks.workunit.client.0.smithi071.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/218867,v1:127.0.0.1:6827/218867] [v2:127.0.0.1:6828/218867,v1:127.0.0.1:6829/218867] exists,up b8fbe2eb-a90e-4112-b4de-1f8c84490d2b 2021-07-26T06:43:42.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:43:42.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:43:42.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:43:42.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: create_rbd_pool 2021-07-26T06:43:42.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:43:43.072 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T06:43:43.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:43:43.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:43:43.637 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T06:43:43.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:43:44.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:43:47.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:809: corrupt_and_repair_jerasure: wait_for_clean 2021-07-26T06:43:47.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:43:47.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:43:47.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:43:47.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:43:47.116 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:43:47.117 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:43:47.117 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:43:47.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:43:47.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:43:47.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:43:47.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:43:47.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:43:47.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:43:47.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:43:47.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:43:47.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:43:47.664 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:43:47.664 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:43:47.664 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T06:43:47.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:43:47.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:43:47.665 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:43:47.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836491 2021-07-26T06:43:47.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836491 2021-07-26T06:43:47.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491' 2021-07-26T06:43:47.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:43:47.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:43:47.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672967 2021-07-26T06:43:47.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672967 2021-07-26T06:43:47.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672967' 2021-07-26T06:43:47.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:43:47.871 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:43:47.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509445 2021-07-26T06:43:47.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509445 2021-07-26T06:43:47.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672967 2-64424509445' 2021-07-26T06:43:47.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:43:47.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:43:48.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345924 2021-07-26T06:43:48.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345924 2021-07-26T06:43:48.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672967 2-64424509445 3-85899345924' 2021-07-26T06:43:48.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:43:48.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836491 2021-07-26T06:43:48.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:43:48.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:43:48.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836491 2021-07-26T06:43:48.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:43:48.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836491 2021-07-26T06:43:48.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-07-26T06:43:48.109 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836491 2021-07-26T06:43:48.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:43:48.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836491 2021-07-26T06:43:48.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:43:49.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:43:49.374 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:43:49.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836491 -lt 21474836491 2021-07-26T06:43:49.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:43:49.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672967 2021-07-26T06:43:49.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:43:49.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:43:49.654 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672967 2021-07-26T06:43:49.655 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:43:49.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672967 2021-07-26T06:43:49.656 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672967 2021-07-26T06:43:49.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2021-07-26T06:43:49.657 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:43:49.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672967 -lt 42949672967 2021-07-26T06:43:49.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:43:49.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509445 2021-07-26T06:43:49.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:43:49.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:43:49.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509445 2021-07-26T06:43:49.926 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:43:49.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509445 2021-07-26T06:43:49.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2021-07-26T06:43:49.927 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509445 2021-07-26T06:43:49.927 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:43:50.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509445 -lt 64424509445 2021-07-26T06:43:50.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:43:50.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345924 2021-07-26T06:43:50.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:43:50.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:43:50.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345924 2021-07-26T06:43:50.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:43:50.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345924 2021-07-26T06:43:50.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345924' 2021-07-26T06:43:50.280 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.3 seq 85899345924 2021-07-26T06:43:50.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:43:50.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345924 -lt 85899345924 2021-07-26T06:43:50.547 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:43:50.548 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:43:50.548 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:43:50.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:43:50.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:43:50.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:43:50.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:43:50.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:43:50.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:43:50.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:43:50.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:43:51.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:43:51.169 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:43:51.169 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:43:51.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:43:51.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:43:51.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:43:51.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:43:51.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2021-07-26T06:43:51.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=ecpool 2021-07-26T06:43:51.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T06:43:51.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=true 2021-07-26T06:43:51.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T06:43:51.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2021-07-26T06:43:51.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2021-07-26T06:43:51.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2021-07-26T06:43:52.671 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' created 2021-07-26T06:43:52.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:43:53.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' true = true ']' 2021-07-26T06:43:53.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2021-07-26T06:43:54.338 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 2 allow_ec_overwrites to true 2021-07-26T06:43:54.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T06:43:54.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:43:54.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:43:54.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:43:54.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:43:54.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:43:54.351 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:43:54.351 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:43:54.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:43:54.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:43:54.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:43:54.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:43:54.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:43:54.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:43:54.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:43:54.446 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:43:54.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:43:54.714 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:43:54.714 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:43:54.715 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T06:43:54.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:43:54.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:43:54.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:43:54.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836493 2021-07-26T06:43:54.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836493 2021-07-26T06:43:54.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836493' 2021-07-26T06:43:54.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:43:54.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:43:54.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672970 2021-07-26T06:43:54.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672970 2021-07-26T06:43:54.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836493 1-42949672970' 2021-07-26T06:43:54.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:43:54.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:43:55.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509447 2021-07-26T06:43:55.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509447 2021-07-26T06:43:55.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836493 1-42949672970 2-64424509447' 2021-07-26T06:43:55.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:43:55.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:43:55.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345926 2021-07-26T06:43:55.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345926 2021-07-26T06:43:55.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836493 1-42949672970 2-64424509447 3-85899345926' 2021-07-26T06:43:55.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:43:55.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836493 2021-07-26T06:43:55.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:43:55.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:43:55.354 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836493 2021-07-26T06:43:55.354 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:43:55.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836493 2021-07-26T06:43:55.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2021-07-26T06:43:55.355 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836493 2021-07-26T06:43:55.355 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:43:55.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836492 -lt 21474836493 2021-07-26T06:43:55.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:43:56.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:43:56.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:43:56.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836493 -lt 21474836493 2021-07-26T06:43:56.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:43:56.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672970 2021-07-26T06:43:56.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:43:56.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:43:56.905 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672970 2021-07-26T06:43:56.905 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:43:56.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672970 2021-07-26T06:43:56.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2021-07-26T06:43:56.906 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672970 2021-07-26T06:43:56.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:43:57.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672970 -lt 42949672970 2021-07-26T06:43:57.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:43:57.287 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509447 2021-07-26T06:43:57.287 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:43:57.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:43:57.289 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509447 2021-07-26T06:43:57.289 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:43:57.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509447 2021-07-26T06:43:57.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2021-07-26T06:43:57.290 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509447 2021-07-26T06:43:57.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:43:57.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509448 -lt 64424509447 2021-07-26T06:43:57.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:43:57.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345926 2021-07-26T06:43:57.593 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:43:57.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:43:57.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345926 2021-07-26T06:43:57.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:43:57.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345926 2021-07-26T06:43:57.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2021-07-26T06:43:57.595 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.3 seq 85899345926 2021-07-26T06:43:57.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:43:57.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345926 -lt 85899345926 2021-07-26T06:43:57.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:43:57.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:43:57.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:43:58.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:43:58.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:43:58.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:43:58.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:43:58.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:43:58.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:43:58.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:43:58.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:43:58.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:43:58.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:43:58.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:43:58.445 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:43:58.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:43:58.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:43:58.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:43:58.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T06:43:58.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:812: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2021-07-26T06:43:58.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2021-07-26T06:43:58.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local poolname=ecpool 2021-07-26T06:43:58.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:250: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2021-07-26T06:43:58.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:43:58.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T06:43:58.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2021-07-26T06:43:58.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:43:58.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:43:58.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:43:59.273 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:43:59.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:43:59.741 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:43:59.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:43:59.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:43:59.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T06:44:00.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2021-07-26T06:44:00.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=ecpool 2021-07-26T06:44:00.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:44:00.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:44:00.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:44:00.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: local primary=3 2021-07-26T06:44:00.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2021-07-26T06:44:00.430 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2021-07-26T06:44:00.431 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: sed -e s/3// 2021-07-26T06:44:00.431 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=ecpool 2021-07-26T06:44:00.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=SOMETHING 2021-07-26T06:44:00.432 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:44:00.432 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T06:44:00.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=3 2021-07-26T06:44:00.721 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:44:00.722 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:44:00.722 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T06:44:00.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 3 1 2 0 2021-07-26T06:44:00.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local -a osds 2021-07-26T06:44:00.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local not_primary_first=1 2021-07-26T06:44:00.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_second=2 2021-07-26T06:44:00.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2021-07-26T06:44:00.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2021-07-26T06:44:00.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2021-07-26T06:44:00.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=3 2021-07-26T06:44:00.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:44:00.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:44:00.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:44:00.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:44:00.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:44:00.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:44:00.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:44:00.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:44:00.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:44:00.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:44:00.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:44:00.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:44:00.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:44:00.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:44:00.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:44:00.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:44:00.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:44:00.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:44:00.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:44:00.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:44:00.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:44:00.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:44:00.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:44:00.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2021-07-26T06:44:03.298 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:44:03.298 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:44:03.299 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:44:03.299 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2021-07-26T06:44:03.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:44:03.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:44:03.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:44:03.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:44:03.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:44:03.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:44:03.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:44:03.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:44:03.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:44:03.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:44:03.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:44:03.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:44:03.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:44:03.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:44:03.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:44:03.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:44:03.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:44:03.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:44:03.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:44:03.825 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:44:03.825 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:44:03.826 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:44:03.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:44:03.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:44:03.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:44:03.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:44:03.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:44:03.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:44:03.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:44:03.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:44:03.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:44:03.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:44:03.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:44:03.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:44:03.828 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.3 2021-07-26T06:44:03.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:44:03.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:44:03.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:44:03.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:44:03.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:44:03.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:44:03.845 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:03.847+0000 7f524bd280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:44:03.848 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:03.849+0000 7f524bd280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:44:03.849 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:03.850+0000 7f524bd280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:44:04.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:44:04.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:44:04.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:44:04.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:44:04.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:44:04.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:04.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:44:04.103 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:44:04.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:04.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:44:04.366 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:04.367+0000 7f524bd280c0 -1 Falling back to public interface 2021-07-26T06:44:04.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:44:05.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:44:05.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:05.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:44:05.370 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:44:05.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:05.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:44:05.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:44:06.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:44:06.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:06.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:44:06.648 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:44:06.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:06.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:44:06.879 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:06.880+0000 7f524bd280c0 -1 osd.3 37 log_to_monitors {default=true} 2021-07-26T06:44:06.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:44:07.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:44:07.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:07.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:44:07.928 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:44:07.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:07.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:44:08.208 INFO:tasks.workunit.client.0.smithi071.stderr:osd.3 up in weight 1 up_from 42 up_thru 42 down_at 38 last_clean_interval [20,37) [v2:127.0.0.1:6826/220959,v1:127.0.0.1:6827/220959] [v2:127.0.0.1:6828/220959,v1:127.0.0.1:6829/220959] exists,up b8fbe2eb-a90e-4112-b4de-1f8c84490d2b 2021-07-26T06:44:08.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:44:08.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:44:08.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:44:08.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:44:08.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:44:08.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:44:08.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:44:08.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:44:08.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:44:08.211 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:44:08.211 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:44:08.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:44:08.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:44:08.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:44:08.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:44:08.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:44:08.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:44:08.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:44:08.304 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:44:08.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:44:08.702 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:44:08.702 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:44:08.702 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T06:44:08.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:44:08.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:08.703 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:44:08.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836497 2021-07-26T06:44:08.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836497 2021-07-26T06:44:08.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836497' 2021-07-26T06:44:08.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:08.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:44:08.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672973 2021-07-26T06:44:08.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672973 2021-07-26T06:44:08.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836497 1-42949672973' 2021-07-26T06:44:08.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:08.979 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:44:09.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509451 2021-07-26T06:44:09.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509451 2021-07-26T06:44:09.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836497 1-42949672973 2-64424509451' 2021-07-26T06:44:09.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:09.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:44:09.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=180388626434 2021-07-26T06:44:09.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 180388626434 2021-07-26T06:44:09.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836497 1-42949672973 2-64424509451 3-180388626434' 2021-07-26T06:44:09.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:09.216 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836497 2021-07-26T06:44:09.216 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:09.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:44:09.217 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836497 2021-07-26T06:44:09.217 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:09.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836497 2021-07-26T06:44:09.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2021-07-26T06:44:09.219 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836497 2021-07-26T06:44:09.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:44:09.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836496 -lt 21474836497 2021-07-26T06:44:09.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:44:10.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:44:10.483 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:44:10.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836496 -lt 21474836497 2021-07-26T06:44:10.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:44:11.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:44:11.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:44:12.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836497 -lt 21474836497 2021-07-26T06:44:12.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:12.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672973 2021-07-26T06:44:12.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:12.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:44:12.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672973 2021-07-26T06:44:12.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:12.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672973 2021-07-26T06:44:12.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2021-07-26T06:44:12.041 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672973 2021-07-26T06:44:12.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:44:12.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672974 -lt 42949672973 2021-07-26T06:44:12.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:12.308 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509451 2021-07-26T06:44:12.309 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:12.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:44:12.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509451 2021-07-26T06:44:12.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:12.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509451 2021-07-26T06:44:12.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2021-07-26T06:44:12.311 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509451 2021-07-26T06:44:12.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:44:12.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509451 -lt 64424509451 2021-07-26T06:44:12.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:12.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-180388626434 2021-07-26T06:44:12.620 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:12.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:44:12.620 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-180388626434 2021-07-26T06:44:12.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:12.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=180388626434 2021-07-26T06:44:12.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 180388626434' 2021-07-26T06:44:12.622 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 180388626434 2021-07-26T06:44:12.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:44:12.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 180388626435 -lt 180388626434 2021-07-26T06:44:12.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:44:12.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:12.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:13.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:44:13.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:13.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:13.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:13.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:13.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:13.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:13.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:13.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:44:13.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:13.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:13.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:13.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:44:13.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:44:13.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:44:13.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2021-07-26T06:44:13.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:44:13.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:44:13.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:44:13.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:44:14.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2021-07-26T06:44:14.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2021-07-26T06:44:14.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:44:14.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:44:14.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:44:14.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:44:14.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:44:14.169 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:44:14.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:43:52.526965+0000 2021-07-26T06:44:14.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:44:14.707 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:44:14.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:43:52.526965+0000 2021-07-26T06:44:14.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:44:14.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:43:52.526965+0000 2021-07-26T06:44:14.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:44:14.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:44:14.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:44:14.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:44:14.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:44:14.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:44:14.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:44:14.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:44:14.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:43:52.526965+0000 '>' 2021-07-26T06:43:52.526965+0000 2021-07-26T06:44:14.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:44:15.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:44:15.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:44:15.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:44:15.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:44:15.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:44:15.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:44:15.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:44:16.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:43:52.526965+0000 '>' 2021-07-26T06:43:52.526965+0000 2021-07-26T06:44:16.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:44:17.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:44:17.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:44:17.255 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:44:17.255 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:44:17.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:44:17.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:44:17.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:44:17.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:43:52.526965+0000 '>' 2021-07-26T06:43:52.526965+0000 2021-07-26T06:44:17.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:44:18.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:44:18.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:44:18.530 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:44:18.531 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:44:18.531 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:44:18.531 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:44:18.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:44:18.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:14.874397+0000 '>' 2021-07-26T06:43:52.526965+0000 2021-07-26T06:44:18.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:44:18.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:44:18.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:44:18.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:44:18.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:44:18.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:44:18.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:44:18.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:44:18.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:44:18.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:44:18.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:44:18.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:44:18.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:44:18.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:44:18.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:44:18.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:44:18.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:44:18.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:44:18.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:44:18.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:44:18.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:44:18.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:44:18.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:44:18.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:44:18.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2021-07-26T06:44:20.860 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:44:20.860 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2021-07-26T06:44:20.861 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:44:20.861 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:44:20.861 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:44:20.862 INFO:tasks.workunit.client.0.smithi071.stdout:hinfo_key 2021-07-26T06:44:20.862 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:44:21.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:44:21.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:44:21.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:44:21.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:44:21.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:44:21.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:44:21.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:44:21.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:44:21.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:44:21.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:44:21.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:44:21.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:44:21.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:44:21.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:44:21.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:44:21.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:44:21.136 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:44:21.136 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:44:21.136 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:44:21.136 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:44:21.137 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:44:21.137 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:44:21.137 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:44:21.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:44:21.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:44:21.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:44:21.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:44:21.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:44:21.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:44:21.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:44:21.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:44:21.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:44:21.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:44:21.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:44:21.140 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.3 2021-07-26T06:44:21.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:44:21.142 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:44:21.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:44:21.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:44:21.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:44:21.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:44:21.157 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:21.158+0000 7f41022a60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:44:21.159 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:21.160+0000 7f41022a60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:44:21.160 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:21.161+0000 7f41022a60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:44:21.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:44:21.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:44:21.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:44:21.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:44:21.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:44:21.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:21.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:44:21.416 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:44:21.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:21.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:44:21.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:44:21.688 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:21.690+0000 7f41022a60c0 -1 Falling back to public interface 2021-07-26T06:44:22.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:44:22.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:22.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:44:22.684 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:44:22.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:22.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:44:22.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:44:23.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:44:23.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:23.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:44:23.964 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:44:23.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:23.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:44:24.158 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:24.159+0000 7f41022a60c0 -1 osd.3 43 log_to_monitors {default=true} 2021-07-26T06:44:24.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:44:25.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:44:25.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:25.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:44:25.272 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:44:25.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:25.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:44:25.655 INFO:tasks.workunit.client.0.smithi071.stderr:osd.3 up in weight 1 up_from 48 up_thru 48 down_at 44 last_clean_interval [42,43) [v2:127.0.0.1:6826/222567,v1:127.0.0.1:6827/222567] [v2:127.0.0.1:6828/222567,v1:127.0.0.1:6829/222567] exists,up b8fbe2eb-a90e-4112-b4de-1f8c84490d2b 2021-07-26T06:44:25.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:44:25.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:44:25.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:44:25.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:44:25.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:44:25.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:44:25.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:44:25.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:44:25.657 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:44:25.658 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:44:25.658 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:44:25.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:44:25.659 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:44:25.659 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:44:25.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:44:25.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:44:25.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:44:25.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:44:25.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:44:26.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:44:26.140 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:44:26.141 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:44:26.141 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T06:44:26.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:44:26.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:26.142 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:44:26.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836501 2021-07-26T06:44:26.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836501 2021-07-26T06:44:26.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836501' 2021-07-26T06:44:26.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:26.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:44:26.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672978 2021-07-26T06:44:26.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672978 2021-07-26T06:44:26.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836501 1-42949672978' 2021-07-26T06:44:26.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:26.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:44:26.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509456 2021-07-26T06:44:26.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509456 2021-07-26T06:44:26.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836501 1-42949672978 2-64424509456' 2021-07-26T06:44:26.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:26.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:44:26.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=206158430210 2021-07-26T06:44:26.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 206158430210 2021-07-26T06:44:26.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836501 1-42949672978 2-64424509456 3-206158430210' 2021-07-26T06:44:26.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:26.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836501 2021-07-26T06:44:26.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:26.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:44:26.724 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836501 2021-07-26T06:44:26.724 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:26.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836501 2021-07-26T06:44:26.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2021-07-26T06:44:26.725 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836501 2021-07-26T06:44:26.725 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:44:26.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836501 -lt 21474836501 2021-07-26T06:44:26.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:26.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672978 2021-07-26T06:44:26.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:26.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:44:26.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672978 2021-07-26T06:44:26.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:26.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672978 2021-07-26T06:44:26.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2021-07-26T06:44:26.994 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672978 2021-07-26T06:44:26.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:44:27.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672978 -lt 42949672978 2021-07-26T06:44:27.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:27.273 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509456 2021-07-26T06:44:27.273 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:27.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:44:27.274 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509456 2021-07-26T06:44:27.274 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:27.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509456 2021-07-26T06:44:27.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2021-07-26T06:44:27.276 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509456 2021-07-26T06:44:27.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:44:27.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509456 -lt 64424509456 2021-07-26T06:44:27.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:27.543 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-206158430210 2021-07-26T06:44:27.543 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:27.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:44:27.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-206158430210 2021-07-26T06:44:27.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:27.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=206158430210 2021-07-26T06:44:27.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 206158430210' 2021-07-26T06:44:27.546 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 206158430210 2021-07-26T06:44:27.547 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:44:27.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 206158430210 2021-07-26T06:44:27.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:44:28.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:44:28.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:44:29.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 206158430210 -lt 206158430210 2021-07-26T06:44:29.172 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:44:29.172 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:29.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:29.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:44:29.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:29.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:29.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:29.516 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:29.516 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:29.516 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:29.516 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:29.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:44:29.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:29.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:29.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:30.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:44:30.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:44:30.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:44:30.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:44:35.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:44:35.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2021-07-26T06:44:35.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2021-07-26T06:44:35.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2021-07-26T06:44:35.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=1 2021-07-26T06:44:35.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:44:35.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:44:35.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:44:35.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:44:35.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:44:35.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:44:35.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:44:35.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:44:35.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:44:35.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:44:35.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:44:35.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:44:35.886 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:44:35.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:44:35.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:44:35.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:44:35.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:44:35.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:44:35.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:44:35.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:44:35.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:44:35.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:44:35.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:44:35.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2021-07-26T06:44:38.197 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:44:38.197 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:44:38.197 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:44:38.199 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:44:38.199 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:44:38.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:44:38.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:44:38.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:44:38.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:44:38.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:44:38.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:44:38.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:44:38.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:44:38.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:44:38.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:44:38.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:44:38.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:44:38.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:44:38.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:44:38.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:44:38.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:44:38.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:44:38.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:44:38.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:44:38.728 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:44:38.728 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:44:38.728 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:44:38.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:44:38.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:44:38.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:44:38.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:44:38.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:44:38.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:44:38.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:44:38.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:44:38.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:44:38.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:44:38.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:44:38.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:44:38.731 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:44:38.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:44:38.839 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:44:38.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:44:38.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:44:38.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:44:38.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:44:38.841 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:38.747+0000 7f69c167f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:44:38.842 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:38.749+0000 7f69c167f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:44:38.842 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:38.750+0000 7f69c167f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:44:39.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:44:39.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:44:39.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:44:39.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:44:39.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:44:39.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:39.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:44:39.004 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:44:39.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:39.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:44:39.268 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:39.268+0000 7f69c167f0c0 -1 Falling back to public interface 2021-07-26T06:44:39.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:44:40.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:44:40.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:40.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:44:40.278 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:44:40.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:40.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:44:40.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:44:41.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:44:41.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:41.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:44:41.556 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:44:41.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:41.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:44:41.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:44:42.062 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:42.062+0000 7f69c167f0c0 -1 osd.1 50 log_to_monitors {default=true} 2021-07-26T06:44:42.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:44:42.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:42.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:44:42.831 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:44:42.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:42.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:44:43.285 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 51 last_clean_interval [10,50) [v2:127.0.0.1:6810/223988,v1:127.0.0.1:6811/223988] [v2:127.0.0.1:6812/223988,v1:127.0.0.1:6813/223988] exists,up 6a7f7fc8-4b60-487d-84b4-acde9f69054f 2021-07-26T06:44:43.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:44:43.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:44:43.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:44:43.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:44:43.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:44:43.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:44:43.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:44:43.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:44:43.288 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:44:43.288 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:44:43.288 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:44:43.289 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:44:43.289 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:44:43.289 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:44:43.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:44:43.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:44:43.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:44:43.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:44:43.382 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:44:43.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:44:43.770 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:44:43.770 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:44:43.771 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T06:44:43.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:44:43.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:43.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:44:43.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836506 2021-07-26T06:44:43.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836506 2021-07-26T06:44:43.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506' 2021-07-26T06:44:43.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:43.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:44:44.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=236223201282 2021-07-26T06:44:44.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 236223201282 2021-07-26T06:44:44.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-236223201282' 2021-07-26T06:44:44.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:44.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:44:44.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509460 2021-07-26T06:44:44.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509460 2021-07-26T06:44:44.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-236223201282 2-64424509460' 2021-07-26T06:44:44.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:44.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:44:44.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=206158430214 2021-07-26T06:44:44.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 206158430214 2021-07-26T06:44:44.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-236223201282 2-64424509460 3-206158430214' 2021-07-26T06:44:44.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:44.343 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836506 2021-07-26T06:44:44.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:44.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:44:44.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836506 2021-07-26T06:44:44.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:44.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836506 2021-07-26T06:44:44.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2021-07-26T06:44:44.348 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836506 2021-07-26T06:44:44.348 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:44:44.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836505 -lt 21474836506 2021-07-26T06:44:44.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:44:45.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:44:45.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:44:45.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836506 -lt 21474836506 2021-07-26T06:44:45.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:45.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-236223201282 2021-07-26T06:44:45.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:45.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:44:45.892 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-236223201282 2021-07-26T06:44:45.892 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:45.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=236223201282 2021-07-26T06:44:45.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 236223201282' 2021-07-26T06:44:45.893 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 236223201282 2021-07-26T06:44:45.893 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:44:46.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 236223201282 -lt 236223201282 2021-07-26T06:44:46.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:46.199 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509460 2021-07-26T06:44:46.200 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:46.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:44:46.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509460 2021-07-26T06:44:46.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:46.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509460 2021-07-26T06:44:46.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2021-07-26T06:44:46.202 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509460 2021-07-26T06:44:46.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:44:46.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509460 -lt 64424509460 2021-07-26T06:44:46.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:46.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-206158430214 2021-07-26T06:44:46.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:46.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:44:46.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-206158430214 2021-07-26T06:44:46.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:46.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=206158430214 2021-07-26T06:44:46.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 206158430214' 2021-07-26T06:44:46.472 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 206158430214 2021-07-26T06:44:46.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:44:46.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 206158430215 -lt 206158430214 2021-07-26T06:44:46.738 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:44:46.738 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:46.738 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:47.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:44:47.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:47.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:47.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:47.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:47.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:47.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:47.088 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:47.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:44:47.347 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:47.348 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:47.348 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:47.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 5 2021-07-26T06:44:47.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 4 '!=' -1 2021-07-26T06:44:47.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:44:47.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=4 2021-07-26T06:44:47.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:47.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:47.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:47.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:47.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:47.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:47.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:47.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:47.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:47.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:48.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:44:48.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:48.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:48.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:48.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 5 2021-07-26T06:44:48.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 4 '!=' 4 2021-07-26T06:44:48.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:48.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:48.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:48.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:48.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:48.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:48.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:48.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:44:48.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:44:48.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 1 >= 13 )) 2021-07-26T06:44:48.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:48.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.2 2021-07-26T06:44:48.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:48.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:48.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:48.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:48.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:48.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:48.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:48.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:49.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:44:49.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:49.174 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:49.174 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:49.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:44:49.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:44:49.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:44:49.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2021-07-26T06:44:49.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:44:49.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:44:49.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:44:49.500 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:44:49.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2021-07-26T06:44:49.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2021-07-26T06:44:49.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:44:49.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:44:49.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:44:49.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:44:49.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:44:49.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:44:50.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:44:14.874397+0000 2021-07-26T06:44:50.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:44:50.281 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:44:50.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:44:14.874397+0000 2021-07-26T06:44:50.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:44:50.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:44:14.874397+0000 2021-07-26T06:44:50.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:44:50.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:44:50.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:44:50.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:44:50.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:44:50.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:44:50.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:44:50.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:44:50.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:14.874397+0000 '>' 2021-07-26T06:44:14.874397+0000 2021-07-26T06:44:50.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:44:51.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:44:51.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:44:51.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:44:51.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:44:51.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:44:51.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:44:51.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:44:51.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:14.874397+0000 '>' 2021-07-26T06:44:14.874397+0000 2021-07-26T06:44:51.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:44:52.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:44:52.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:44:52.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:44:52.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:44:52.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:44:52.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:44:52.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:44:53.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:14.874397+0000 '>' 2021-07-26T06:44:14.874397+0000 2021-07-26T06:44:53.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:44:54.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:44:54.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:44:54.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:44:54.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:44:54.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:44:54.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:44:54.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:44:54.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:14.874397+0000 '>' 2021-07-26T06:44:14.874397+0000 2021-07-26T06:44:54.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:44:55.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:44:55.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:44:55.409 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:44:55.410 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:44:55.410 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:44:55.410 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:44:55.410 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:44:55.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:51.198066+0000 '>' 2021-07-26T06:44:14.874397+0000 2021-07-26T06:44:55.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:44:55.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:44:55.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:44:55.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:44:55.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:44:55.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:44:55.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:44:55.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:44:55.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:44:55.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:44:55.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:44:55.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:44:55.687 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:44:55.687 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:44:55.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:44:55.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:44:55.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:44:55.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:44:55.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:44:55.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:44:55.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:44:55.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:44:55.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:44:55.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:44:55.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2021-07-26T06:44:57.765 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:44:57.766 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2021-07-26T06:44:57.766 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:44:57.766 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:44:57.767 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:44:57.767 INFO:tasks.workunit.client.0.smithi071.stdout:hinfo_key 2021-07-26T06:44:57.767 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:44:58.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:44:58.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:44:58.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:44:58.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:44:58.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:44:58.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:44:58.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:44:58.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:44:58.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:44:58.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:44:58.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:44:58.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:44:58.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:44:58.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:44:58.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:44:58.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:44:58.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:44:58.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:44:58.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:44:58.291 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:44:58.291 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:44:58.291 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:44:58.292 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:44:58.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:44:58.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:44:58.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:44:58.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:44:58.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:44:58.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:44:58.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:44:58.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:44:58.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:44:58.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:44:58.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:44:58.295 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.3 2021-07-26T06:44:58.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:44:58.295 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:44:58.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:44:58.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:44:58.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:44:58.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:44:58.310 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:58.312+0000 7effca6c80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:44:58.312 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:58.314+0000 7effca6c80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:44:58.313 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:58.315+0000 7effca6c80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:44:58.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:44:58.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:44:58.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:44:58.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:44:58.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:44:58.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:58.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:44:58.566 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:44:58.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:58.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:44:58.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:44:58.840 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:44:58.842+0000 7effca6c80c0 -1 Falling back to public interface 2021-07-26T06:44:59.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:44:59.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:44:59.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:44:59.836 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:44:59.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:44:59.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:45:00.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:01.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:01.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:01.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:45:01.104 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:45:01.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:01.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:45:01.396 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:45:01.397+0000 7effca6c80c0 -1 osd.3 57 log_to_monitors {default=true} 2021-07-26T06:45:01.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:02.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:02.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:02.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:45:02.421 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:45:02.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:02.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:45:02.744 INFO:tasks.workunit.client.0.smithi071.stderr:osd.3 up in weight 1 up_from 62 up_thru 62 down_at 58 last_clean_interval [48,57) [v2:127.0.0.1:6826/225717,v1:127.0.0.1:6827/225717] [v2:127.0.0.1:6828/225717,v1:127.0.0.1:6829/225717] exists,up b8fbe2eb-a90e-4112-b4de-1f8c84490d2b 2021-07-26T06:45:02.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:45:02.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:45:02.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:45:02.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:45:02.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:45:02.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:45:02.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:45:02.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:45:02.747 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:45:02.747 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:45:02.748 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:45:02.748 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:45:02.748 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:45:02.748 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:45:02.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:45:02.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:45:02.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:45:02.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:45:02.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:45:03.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:45:03.189 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:45:03.189 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:45:03.189 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T06:45:03.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:45:03.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:03.190 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:45:03.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836511 2021-07-26T06:45:03.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836511 2021-07-26T06:45:03.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836511' 2021-07-26T06:45:03.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:03.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:45:03.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=236223201287 2021-07-26T06:45:03.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 236223201287 2021-07-26T06:45:03.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836511 1-236223201287' 2021-07-26T06:45:03.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:03.517 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:45:03.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509465 2021-07-26T06:45:03.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509465 2021-07-26T06:45:03.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836511 1-236223201287 2-64424509465' 2021-07-26T06:45:03.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:03.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:45:03.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=266287972354 2021-07-26T06:45:03.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 266287972354 2021-07-26T06:45:03.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836511 1-236223201287 2-64424509465 3-266287972354' 2021-07-26T06:45:03.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:03.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836511 2021-07-26T06:45:03.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:03.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:45:03.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836511 2021-07-26T06:45:03.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:03.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836511 2021-07-26T06:45:03.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2021-07-26T06:45:03.755 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836511 2021-07-26T06:45:03.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:45:04.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836510 -lt 21474836511 2021-07-26T06:45:04.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:45:05.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:45:05.020 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:45:05.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836511 -lt 21474836511 2021-07-26T06:45:05.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:05.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-236223201287 2021-07-26T06:45:05.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:05.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:45:05.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-236223201287 2021-07-26T06:45:05.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:05.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=236223201287 2021-07-26T06:45:05.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 236223201287' 2021-07-26T06:45:05.314 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 236223201287 2021-07-26T06:45:05.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:45:05.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 236223201287 -lt 236223201287 2021-07-26T06:45:05.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:05.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509465 2021-07-26T06:45:05.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:05.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:45:05.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509465 2021-07-26T06:45:05.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:05.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509465 2021-07-26T06:45:05.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2021-07-26T06:45:05.582 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509465 2021-07-26T06:45:05.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:45:05.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509465 -lt 64424509465 2021-07-26T06:45:05.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:05.848 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-266287972354 2021-07-26T06:45:05.848 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:05.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:45:05.856 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-266287972354 2021-07-26T06:45:05.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:05.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=266287972354 2021-07-26T06:45:05.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 266287972354' 2021-07-26T06:45:05.858 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 266287972354 2021-07-26T06:45:05.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:45:06.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 266287972354 -lt 266287972354 2021-07-26T06:45:06.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:45:06.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:06.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:06.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:45:06.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:06.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:06.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:06.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:06.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:06.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:06.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:06.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:45:06.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:06.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:06.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:07.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:45:07.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:45:07.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:45:07.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:45:13.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:45:13.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2021-07-26T06:45:13.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2021-07-26T06:45:13.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2021-07-26T06:45:13.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=1 2021-07-26T06:45:13.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=2 2021-07-26T06:45:13.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2021-07-26T06:45:13.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:45:13.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:45:13.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:45:13.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 226729"' 2021-07-26T06:45:13.375 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 226729' 2021-07-26T06:45:13.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T06:45:13.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:45:13.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:45:13.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:45:13.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 226731"' 2021-07-26T06:45:13.377 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 226731' 2021-07-26T06:45:13.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2021-07-26T06:45:13.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/226732: /' 2021-07-26T06:45:13.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 226729 226731' 2021-07-26T06:45:13.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:45:13.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:45:13.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 226729 2021-07-26T06:45:13.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T06:45:13.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/226735: /' 2021-07-26T06:45:15.846 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:45:15.846 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:45:15.847 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:45:15.847 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:45:15.847 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:45:15.847 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:45:15.848 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:45:15.848 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:45:15.848 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:45:15.848 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:45:15.849 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:45:15.849 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:45:15.849 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:45:15.849 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:45:15.850 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:45:15.850 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:45:15.851 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:45:15.851 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:45:15.851 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:45:15.851 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:45:15.852 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:45:15.852 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:45:15.852 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2021-07-26T06:45:15.852 INFO:tasks.workunit.client.0.smithi071.stderr:226732: Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:45:15.852 INFO:tasks.workunit.client.0.smithi071.stderr:226732: Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:45:15.853 INFO:tasks.workunit.client.0.smithi071.stderr:226732: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:45:15.853 INFO:tasks.workunit.client.0.smithi071.stderr:226732: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:45:15.853 INFO:tasks.workunit.client.0.smithi071.stderr:226732: remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:45:15.853 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:45:15.853 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:45:15.854 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:45:15.854 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:45:15.854 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:45:15.854 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:45:15.855 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:45:15.855 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:45:15.856 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:45:15.856 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:45:15.856 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:45:15.856 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:45:15.857 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:45:15.857 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:45:15.857 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:45:15.857 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:45:15.857 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:45:15.858 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:45:15.858 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:45:15.858 INFO:tasks.workunit.client.0.smithi071.stderr:226732: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:45:15.859 INFO:tasks.workunit.client.0.smithi071.stderr:226732: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:45:15.859 INFO:tasks.workunit.client.0.smithi071.stderr:226732: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:45:15.859 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:45:15.859 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:45:15.860 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:45:15.860 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:45:15.860 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:45:15.860 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:45:15.860 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:45:15.861 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:45:15.861 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:45:15.861 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:45:15.861 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:45:15.861 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:45:15.862 INFO:tasks.workunit.client.0.smithi071.stderr:226732: start osd.1 2021-07-26T06:45:16.122 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-di226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:45:16.122 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:45:16.123 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=2 2021-07-26T06:45:16.123 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:45:16.123 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T06:45:16.123 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:45:16.124 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:45:16.124 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=2 2021-07-26T06:45:16.124 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:45:16.124 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2021-07-26T06:45:16.125 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:45:16.125 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:45:16.125 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:45:16.125 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:45:16.125 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:45:16.126 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:45:16.126 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T06:45:16.126 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:45:16.126 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:45:16.127 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=2 2021-07-26T06:45:16.127 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:45:16.127 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:45:16.127 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2021-07-26T06:45:16.127 INFO:tasks.workunit.client.0.smithi071.stderr:226735: Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:45:16.128 INFO:tasks.workunit.client.0.smithi071.stderr:226735: Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T06:45:16.128 INFO:tasks.workunit.client.0.smithi071.stderr:226735: Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:45:16.128 INFO:tasks.workunit.client.0.smithi071.stderr:226735: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:45:16.128 INFO:tasks.workunit.client.0.smithi071.stderr:226735: remove 2#2:eb822e21:::SOMETHING:head# 2021-07-26T06:45:16.129 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2021-07-26T06:45:16.129 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:45:16.129 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:45:16.129 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:45:16.129 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:45:16.130 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:45:16.130 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:45:16.131 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:45:16.131 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:45:16.131 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:45:16.132 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:45:16.132 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:45:16.132 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:45:16.132 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:45:16.132 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:45:16.133 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:45:16.133 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:45:16.133 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:45:16.133 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:45:16.134 INFO:tasks.workunit.client.0.smithi071.stderr:226735: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:45:16.134 INFO:tasks.workunit.client.0.smithi071.stderr:226735: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:45:16.134 INFO:tasks.workunit.client.0.smithi071.stderr:226735: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:45:16.134 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:45:16.135 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:45:16.135 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:45:16.135 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:45:16.135 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:45:16.136 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:45:16.136 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:45:16.136 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:45:16.136 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:45:16.136 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:45:16.137 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:45:16.137 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:45:16.137 INFO:tasks.workunit.client.0.smithi071.stderr:226735: start osd.2 2021-07-26T06:45:19.981 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:45:19.982 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:45:19.982 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:45:19.983 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:45:19.983 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:45:19.983 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:45:19.983 INFO:tasks.workunit.client.0.smithi071.stderr:226732: 2021-07-26T06:45:15.875+0000 7fed6cc1c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:19.984 INFO:tasks.workunit.client.0.smithi071.stderr:226732: 2021-07-26T06:45:15.877+0000 7fed6cc1c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:19.984 INFO:tasks.workunit.client.0.smithi071.stderr:226732: 2021-07-26T06:45:15.878+0000 7fed6cc1c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:19.984 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:45:19.985 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:45:19.985 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:45:19.985 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:45:19.986 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:45:19.986 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:19.986 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:45:19.986 INFO:tasks.workunit.client.0.smithi071.stderr:226732: 0 2021-07-26T06:45:19.987 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:19.987 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:45:19.987 INFO:tasks.workunit.client.0.smithi071.stderr:226732: 2021-07-26T06:45:16.397+0000 7fed6cc1c0c0 -1 Falling back to public interface 2021-07-26T06:45:19.988 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:19.988 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:19.988 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:19.988 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:45:19.989 INFO:tasks.workunit.client.0.smithi071.stderr:226732: 1 2021-07-26T06:45:19.989 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:19.989 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:45:19.990 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:19.990 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:19.990 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:19.991 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:45:19.991 INFO:tasks.workunit.client.0.smithi071.stderr:226732: 2 2021-07-26T06:45:19.992 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:19.992 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:45:19.992 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:19.993 INFO:tasks.workunit.client.0.smithi071.stderr:226732: 2021-07-26T06:45:19.096+0000 7fed6cc1c0c0 -1 osd.1 63 log_to_monitors {default=true} 2021-07-26T06:45:19.993 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:19.993 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:20.326 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wr=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:45:20.326 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/2/whoami 2021-07-26T06:45:20.326 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:45:20.327 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:45:20.327 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:45:20.327 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:45:20.328 INFO:tasks.workunit.client.0.smithi071.stderr:226735: 2021-07-26T06:45:16.148+0000 7f22d620b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:20.328 INFO:tasks.workunit.client.0.smithi071.stderr:226735: 2021-07-26T06:45:16.150+0000 7f22d620b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:20.328 INFO:tasks.workunit.client.0.smithi071.stderr:226735: 2021-07-26T06:45:16.151+0000 7f22d620b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:20.328 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:45:20.329 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:45:20.329 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:45:20.329 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:45:20.330 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:45:20.330 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:20.330 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:45:20.331 INFO:tasks.workunit.client.0.smithi071.stderr:226735: 0 2021-07-26T06:45:20.331 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:20.331 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:45:20.332 INFO:tasks.workunit.client.0.smithi071.stderr:226735: 2021-07-26T06:45:16.671+0000 7f22d620b0c0 -1 Falling back to public interface 2021-07-26T06:45:20.332 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:20.332 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:20.333 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:20.333 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:45:20.333 INFO:tasks.workunit.client.0.smithi071.stderr:226735: 1 2021-07-26T06:45:20.333 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:20.334 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:45:20.334 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:20.334 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:20.335 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:20.335 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:45:20.335 INFO:tasks.workunit.client.0.smithi071.stderr:226735: 2 2021-07-26T06:45:20.335 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:20.336 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:45:20.336 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:20.336 INFO:tasks.workunit.client.0.smithi071.stderr:226735: 2021-07-26T06:45:19.700+0000 7f22d620b0c0 -1 osd.2 63 log_to_monitors {default=true} 2021-07-26T06:45:20.337 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:20.337 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:21.167 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:45:21.168 INFO:tasks.workunit.client.0.smithi071.stderr:226732: 3 2021-07-26T06:45:21.168 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:21.168 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:45:21.168 INFO:tasks.workunit.client.0.smithi071.stderr:226732: osd.1 up in weight 1 up_from 68 up_thru 58 down_at 64 last_clean_interval [55,63) [v2:127.0.0.1:6810/227537,v1:127.0.0.1:6811/227537] [v2:127.0.0.1:6812/227537,v1:127.0.0.1:6813/227537] exists,up 6a7f7fc8-4b60-487d-84b4-acde9f69054f 2021-07-26T06:45:21.169 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:45:21.169 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:45:21.169 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:45:21.169 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:45:21.169 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:45:21.170 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:45:21.170 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:45:21.170 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:45:21.170 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:45:21.171 INFO:tasks.workunit.client.0.smithi071.stderr:226732: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:45:21.171 INFO:tasks.workunit.client.0.smithi071.stderr:226732: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:45:21.171 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:45:21.171 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:45:21.171 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:45:21.172 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:45:21.172 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:45:21.172 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:45:21.172 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:45:21.173 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:45:21.173 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:45:21.173 INFO:tasks.workunit.client.0.smithi071.stderr:226732: 1 2021-07-26T06:45:21.173 INFO:tasks.workunit.client.0.smithi071.stderr:226732: 2 2021-07-26T06:45:21.174 INFO:tasks.workunit.client.0.smithi071.stderr:226732: 3' 2021-07-26T06:45:21.174 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:45:21.174 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:21.174 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:45:21.175 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836515 2021-07-26T06:45:21.175 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836515 2021-07-26T06:45:21.175 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515' 2021-07-26T06:45:21.175 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:21.176 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:45:21.176 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=292057776130 2021-07-26T06:45:21.176 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 292057776130 2021-07-26T06:45:22.532 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shait_for_osd: echo 3 2021-07-26T06:45:22.532 INFO:tasks.workunit.client.0.smithi071.stderr:226735: 3 2021-07-26T06:45:22.532 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:22.532 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:45:22.533 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:22.533 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:22.533 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:22.533 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:45:22.534 INFO:tasks.workunit.client.0.smithi071.stderr:226735: 4 2021-07-26T06:45:22.534 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:22.534 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:45:22.534 INFO:tasks.workunit.client.0.smithi071.stderr:226735: osd.2 up in weight 1 up_from 70 up_thru 51 down_at 64 last_clean_interval [15,63) [v2:127.0.0.1:6818/227586,v1:127.0.0.1:6819/227586] [v2:127.0.0.1:6820/227586,v1:127.0.0.1:6821/227586] exists,up a1dc4974-84bf-431c-9de1-8a3eb6ab83ae 2021-07-26T06:45:22.534 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:45:22.535 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:45:22.535 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:45:22.535 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:45:22.536 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:45:22.536 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:45:22.536 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:45:22.536 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:45:22.536 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:45:22.537 INFO:tasks.workunit.client.0.smithi071.stderr:226735: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:45:22.537 INFO:tasks.workunit.client.0.smithi071.stderr:226735: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:45:22.537 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:45:22.537 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:45:22.538 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:45:22.538 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:45:22.538 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:45:22.538 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:45:22.539 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:45:22.539 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:45:22.539 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:45:22.539 INFO:tasks.workunit.client.0.smithi071.stderr:226735: 1 2021-07-26T06:45:22.539 INFO:tasks.workunit.client.0.smithi071.stderr:226735: 2 2021-07-26T06:45:22.540 INFO:tasks.workunit.client.0.smithi071.stderr:226735: 3' 2021-07-26T06:45:22.540 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:45:22.540 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:22.540 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:45:22.541 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836517 2021-07-26T06:45:22.541 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836517 2021-07-26T06:45:23.117 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/ce:2081: flush_pg_stats: seqs=' 0-21474836515 1-292057776130' 2021-07-26T06:45:23.118 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:23.118 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:45:23.118 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=300647710722 2021-07-26T06:45:23.118 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 300647710722 2021-07-26T06:45:23.119 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-292057776130 2-300647710722' 2021-07-26T06:45:23.119 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:23.119 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:45:23.119 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=266287972358 2021-07-26T06:45:23.120 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 266287972358 2021-07-26T06:45:23.120 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-292057776130 2-300647710722 3-266287972358' 2021-07-26T06:45:23.120 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:23.121 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836515 2021-07-26T06:45:23.121 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:23.121 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:45:23.122 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836515 2021-07-26T06:45:23.122 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:23.122 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836515 2021-07-26T06:45:23.123 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2021-07-26T06:45:23.123 INFO:tasks.workunit.client.0.smithi071.stderr:226732: waiting osd.0 seq 21474836515 2021-07-26T06:45:23.123 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:45:23.123 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836514 -lt 21474836515 2021-07-26T06:45:23.123 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:45:23.124 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:45:23.124 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:45:23.124 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836517 -lt 21474836515 2021-07-26T06:45:23.124 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:23.125 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-292057776130 2021-07-26T06:45:23.125 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:23.125 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:45:23.125 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-292057776130 2021-07-26T06:45:23.125 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:23.126 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=292057776130 2021-07-26T06:45:23.126 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2021-07-26T06:45:23.126 INFO:tasks.workunit.client.0.smithi071.stderr:226732: waiting osd.1 seq 292057776130 2021-07-26T06:45:23.199 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_statphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836517' 2021-07-26T06:45:23.199 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:23.200 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:45:23.200 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=292057776131 2021-07-26T06:45:23.200 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 292057776131 2021-07-26T06:45:23.200 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836517 1-292057776131' 2021-07-26T06:45:23.201 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:23.201 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:45:23.201 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=300647710723 2021-07-26T06:45:23.201 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 300647710723 2021-07-26T06:45:23.202 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836517 1-292057776131 2-300647710723' 2021-07-26T06:45:23.202 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:23.202 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:45:23.202 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=266287972360 2021-07-26T06:45:23.203 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 266287972360 2021-07-26T06:45:23.203 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836517 1-292057776131 2-300647710723 3-266287972360' 2021-07-26T06:45:23.203 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:23.203 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836517 2021-07-26T06:45:23.203 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:23.204 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:45:23.204 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836517 2021-07-26T06:45:23.204 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:23.204 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836517 2021-07-26T06:45:23.204 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2021-07-26T06:45:23.205 INFO:tasks.workunit.client.0.smithi071.stderr:226735: waiting osd.0 seq 21474836517 2021-07-26T06:45:23.205 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:45:23.205 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836517 -lt 21474836517 2021-07-26T06:45:23.205 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:23.206 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-292057776131 2021-07-26T06:45:23.206 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:23.206 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:45:23.206 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-292057776131 2021-07-26T06:45:23.206 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:23.207 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=292057776131 2021-07-26T06:45:24.349 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: s: ceph osd last-stat-seq 1 2021-07-26T06:45:24.349 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 292057776130 -lt 292057776130 2021-07-26T06:45:24.349 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:24.350 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-300647710722 2021-07-26T06:45:24.350 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:24.350 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:45:24.350 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-300647710722 2021-07-26T06:45:24.351 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:24.351 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=300647710722 2021-07-26T06:45:24.351 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 300647710722' 2021-07-26T06:45:24.351 INFO:tasks.workunit.client.0.smithi071.stderr:226732: waiting osd.2 seq 300647710722 2021-07-26T06:45:24.352 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:45:24.352 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 300647710722 -lt 300647710722 2021-07-26T06:45:24.352 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:24.352 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-266287972358 2021-07-26T06:45:24.353 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:24.353 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:45:24.353 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-266287972358 2021-07-26T06:45:24.354 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:24.354 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=266287972358 2021-07-26T06:45:24.354 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 266287972358' 2021-07-26T06:45:24.354 INFO:tasks.workunit.client.0.smithi071.stderr:226732: waiting osd.3 seq 266287972358 2021-07-26T06:45:24.355 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:45:24.355 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 266287972359 -lt 266287972358 2021-07-26T06:45:24.356 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:45:24.356 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:24.356 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:24.356 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:45:24.357 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:24.357 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:24.357 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:24.357 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:24.358 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:24.358 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:24.999 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:24.999 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:45:24.999 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:25.000 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:25.000 INFO:tasks.workunit.client.0.smithi071.stderr:226732: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:25.000 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:45:25.000 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:45:25.001 INFO:tasks.workunit.client.0.smithi071.stderr:226732: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:45:25.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:45:25.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:45:25.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 226731 2021-07-26T06:45:25.721 INFO:tasks.workunit.client.0.smithi071.stderr:echo 'waiting osd.1 seq 292057776131' 2021-07-26T06:45:25.721 INFO:tasks.workunit.client.0.smithi071.stderr:226735: waiting osd.1 seq 292057776131 2021-07-26T06:45:25.721 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:45:25.722 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 292057776130 -lt 292057776131 2021-07-26T06:45:25.722 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:45:25.722 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:45:25.722 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:45:25.723 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 292057776132 -lt 292057776131 2021-07-26T06:45:25.723 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:25.723 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-300647710723 2021-07-26T06:45:25.723 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:25.724 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:45:25.724 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-300647710723 2021-07-26T06:45:25.724 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:25.724 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=300647710723 2021-07-26T06:45:25.724 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 300647710723' 2021-07-26T06:45:25.725 INFO:tasks.workunit.client.0.smithi071.stderr:226735: waiting osd.2 seq 300647710723 2021-07-26T06:45:25.725 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:45:25.725 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 300647710723 -lt 300647710723 2021-07-26T06:45:25.725 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:25.726 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-266287972360 2021-07-26T06:45:25.726 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:25.726 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:45:25.727 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-266287972360 2021-07-26T06:45:25.727 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:25.727 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=266287972360 2021-07-26T06:45:25.728 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 266287972360' 2021-07-26T06:45:25.728 INFO:tasks.workunit.client.0.smithi071.stderr:226735: waiting osd.3 seq 266287972360 2021-07-26T06:45:25.728 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:45:25.728 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 266287972360 -lt 266287972360 2021-07-26T06:45:25.729 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:45:25.729 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:25.729 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:25.729 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:45:25.730 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:25.730 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:25.730 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:26.309 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:26.309 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:26.310 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:26.310 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:26.310 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:45:26.311 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:26.311 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:26.312 INFO:tasks.workunit.client.0.smithi071.stderr:226735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:26.312 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:45:26.312 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:45:26.312 INFO:tasks.workunit.client.0.smithi071.stderr:226735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:45:26.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:45:26.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:45:26.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:45:26.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:45:26.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2021-07-26T06:45:26.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:45:26.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2021-07-26T06:45:26.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:45:26.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:45:26.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:45:26.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:45:26.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2021-07-26T06:45:26.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2021-07-26T06:45:26.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:45:26.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:45:26.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:45:26.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:45:26.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:45:26.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:45:26.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:44:51.198066+0000 2021-07-26T06:45:26.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:45:27.103 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:45:27.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:44:51.198066+0000 2021-07-26T06:45:27.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:45:27.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:44:51.198066+0000 2021-07-26T06:45:27.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:45:27.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:45:27.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:45:27.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:45:27.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:45:27.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:45:27.116 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:45:27.116 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:45:27.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:51.198066+0000 '>' 2021-07-26T06:44:51.198066+0000 2021-07-26T06:45:27.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:45:28.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:45:28.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:45:28.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:45:28.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:45:28.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:45:28.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:45:28.416 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:45:28.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:51.198066+0000 '>' 2021-07-26T06:44:51.198066+0000 2021-07-26T06:45:28.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:45:29.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:45:29.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:45:29.686 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:45:29.686 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:45:29.686 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:45:29.686 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:45:29.687 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:45:29.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:51.198066+0000 '>' 2021-07-26T06:44:51.198066+0000 2021-07-26T06:45:29.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:45:30.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:45:30.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:45:30.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:45:30.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:45:30.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:45:30.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:45:30.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:45:31.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:51.198066+0000 '>' 2021-07-26T06:44:51.198066+0000 2021-07-26T06:45:31.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:45:32.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:45:32.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:45:32.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:45:32.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:45:32.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:45:32.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:45:32.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:45:32.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:51.198066+0000 '>' 2021-07-26T06:44:51.198066+0000 2021-07-26T06:45:32.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:45:33.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:45:33.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:45:33.512 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:45:33.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:45:33.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:45:33.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:45:33.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:45:33.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:51.198066+0000 '>' 2021-07-26T06:44:51.198066+0000 2021-07-26T06:45:33.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:45:34.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:45:34.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:45:34.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:45:34.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:45:34.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:45:34.790 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:45:34.790 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:45:35.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:51.198066+0000 '>' 2021-07-26T06:44:51.198066+0000 2021-07-26T06:45:35.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:45:36.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:45:36.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:45:36.061 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:45:36.061 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:45:36.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:45:36.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:45:36.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:45:36.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:51.198066+0000 '>' 2021-07-26T06:44:51.198066+0000 2021-07-26T06:45:36.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:45:37.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:45:37.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:45:37.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:45:37.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:45:37.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:45:37.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:45:37.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:45:37.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:44:51.198066+0000 '>' 2021-07-26T06:44:51.198066+0000 2021-07-26T06:45:37.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:45:38.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:45:38.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:45:38.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:45:38.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:45:38.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:45:38.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:45:38.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:45:38.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:45:32.421576+0000 '>' 2021-07-26T06:44:51.198066+0000 2021-07-26T06:45:38.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:45:38.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2021-07-26T06:45:38.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:45:38.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:45:38.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:45:38.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 229900"' 2021-07-26T06:45:38.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 229900' 2021-07-26T06:45:38.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2021-07-26T06:45:38.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:45:38.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:45:38.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 229901"' 2021-07-26T06:45:38.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 229901' 2021-07-26T06:45:38.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2021-07-26T06:45:38.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 229900 229901' 2021-07-26T06:45:38.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:45:38.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:45:38.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:45:38.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 229900 2021-07-26T06:45:38.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/229903: /' 2021-07-26T06:45:38.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2021-07-26T06:45:38.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/229905: /' 2021-07-26T06:45:41.308 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:45:41.308 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:45:41.309 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=2 2021-07-26T06:45:41.309 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:45:41.309 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2021-07-26T06:45:41.309 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:45:41.310 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:45:41.310 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=2 2021-07-26T06:45:41.310 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:45:41.310 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2021-07-26T06:45:41.311 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:45:41.311 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:45:41.311 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:45:41.311 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:45:41.312 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:45:41.312 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:45:41.312 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2021-07-26T06:45:41.312 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:45:41.313 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:45:41.313 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=2 2021-07-26T06:45:41.313 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:45:41.314 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:45:41.314 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2021-07-26T06:45:41.314 INFO:tasks.workunit.client.0.smithi071.stderr:229905: Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:45:41.314 INFO:tasks.workunit.client.0.smithi071.stderr:229905: Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T06:45:41.315 INFO:tasks.workunit.client.0.smithi071.stderr:229905: Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:45:41.315 INFO:tasks.workunit.client.0.smithi071.stderr:229905: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:45:41.315 INFO:tasks.workunit.client.0.smithi071.stderr:229905: _ 2021-07-26T06:45:41.315 INFO:tasks.workunit.client.0.smithi071.stderr:229905: hinfo_key 2021-07-26T06:45:41.316 INFO:tasks.workunit.client.0.smithi071.stderr:229905: snapset 2021-07-26T06:45:41.316 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2021-07-26T06:45:41.316 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:45:41.317 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:45:41.317 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:45:41.317 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:45:41.317 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:45:41.318 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:45:41.318 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:45:41.318 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:45:41.319 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:45:41.319 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:45:41.319 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:45:41.319 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:45:41.320 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:45:41.320 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:45:41.320 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:45:41.320 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:45:41.321 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:45:41.321 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:45:41.321 INFO:tasks.workunit.client.0.smithi071.stderr:229905: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:45:41.321 INFO:tasks.workunit.client.0.smithi071.stderr:229905: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:45:41.321 INFO:tasks.workunit.client.0.smithi071.stderr:229905: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:45:41.322 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:45:41.322 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:45:41.322 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:45:41.323 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:45:41.323 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:45:41.323 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:45:41.323 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:45:41.324 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:45:41.324 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:45:41.324 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:45:41.324 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:45:41.325 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:45:41.325 INFO:tasks.workunit.client.0.smithi071.stderr:229905: start osd.2 2021-07-26T06:45:41.364 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:45:41.364 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:45:41.365 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:45:41.365 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:45:41.365 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:45:41.365 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:45:41.366 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:45:41.366 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:45:41.366 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:45:41.366 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:45:41.367 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:45:41.367 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:45:41.367 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:45:41.367 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:45:41.367 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:45:41.368 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:45:41.368 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:45:41.368 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:45:41.368 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:45:41.369 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:45:41.369 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:45:41.369 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:45:41.370 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2021-07-26T06:45:41.370 INFO:tasks.workunit.client.0.smithi071.stderr:229903: Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:45:41.370 INFO:tasks.workunit.client.0.smithi071.stderr:229903: Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:45:41.370 INFO:tasks.workunit.client.0.smithi071.stderr:229903: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:45:41.371 INFO:tasks.workunit.client.0.smithi071.stderr:229903: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:45:41.371 INFO:tasks.workunit.client.0.smithi071.stderr:229903: _ 2021-07-26T06:45:41.371 INFO:tasks.workunit.client.0.smithi071.stderr:229903: hinfo_key 2021-07-26T06:45:41.371 INFO:tasks.workunit.client.0.smithi071.stderr:229903: snapset 2021-07-26T06:45:41.372 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:45:41.372 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:45:41.372 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:45:41.372 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:45:41.372 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:45:41.373 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:45:41.373 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:45:41.374 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:45:41.374 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:45:41.374 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:45:41.374 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:45:41.374 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:45:41.375 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:45:41.375 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:45:41.375 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:45:41.375 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:45:41.376 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:45:41.376 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:45:41.376 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:45:41.376 INFO:tasks.workunit.client.0.smithi071.stderr:229903: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:45:41.377 INFO:tasks.workunit.client.0.smithi071.stderr:229903: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:45:41.377 INFO:tasks.workunit.client.0.smithi071.stderr:229903: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:45:41.377 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:45:41.377 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:45:41.378 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:45:41.378 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:45:41.378 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:45:41.378 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:45:41.378 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:45:41.379 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:45:41.379 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:45:41.379 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:45:41.379 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:45:41.380 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:45:41.380 INFO:tasks.workunit.client.0.smithi071.stderr:229903: start osd.1 2021-07-26T06:45:45.464 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:45:45.464 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/2/whoami 2021-07-26T06:45:45.466 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:45:45.466 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:45:45.466 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:45:45.467 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:45:45.467 INFO:tasks.workunit.client.0.smithi071.stderr:229905: 2021-07-26T06:45:41.335+0000 7f493e3840c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:45.467 INFO:tasks.workunit.client.0.smithi071.stderr:229905: 2021-07-26T06:45:41.337+0000 7f493e3840c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:45.467 INFO:tasks.workunit.client.0.smithi071.stderr:229905: 2021-07-26T06:45:41.338+0000 7f493e3840c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:45.467 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:45:45.468 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:45:45.468 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:45:45.468 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:45:45.468 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:45:45.469 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:45.469 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:45:45.469 INFO:tasks.workunit.client.0.smithi071.stderr:229905: 0 2021-07-26T06:45:45.469 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:45.470 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:45:45.470 INFO:tasks.workunit.client.0.smithi071.stderr:229905: 2021-07-26T06:45:41.859+0000 7f493e3840c0 -1 Falling back to public interface 2021-07-26T06:45:45.470 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:45.470 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:45.471 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:45.471 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:45:45.471 INFO:tasks.workunit.client.0.smithi071.stderr:229905: 1 2021-07-26T06:45:45.471 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:45.472 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:45:45.472 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:45.472 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:45.473 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:45.473 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:45:45.473 INFO:tasks.workunit.client.0.smithi071.stderr:229905: 2 2021-07-26T06:45:45.474 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:45.474 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:45:45.474 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:45.474 INFO:tasks.workunit.client.0.smithi071.stderr:229905: 2021-07-26T06:45:44.606+0000 7f493e3840c0 -1 osd.2 71 log_to_monitors {default=true} 2021-07-26T06:45:45.475 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:45.475 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:45.597 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:45:45.597 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:45:45.598 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:45:45.598 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:45:45.598 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:45:45.599 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:45:45.599 INFO:tasks.workunit.client.0.smithi071.stderr:229903: 2021-07-26T06:45:41.390+0000 7f614034b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:45.599 INFO:tasks.workunit.client.0.smithi071.stderr:229903: 2021-07-26T06:45:41.392+0000 7f614034b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:45.599 INFO:tasks.workunit.client.0.smithi071.stderr:229903: 2021-07-26T06:45:41.393+0000 7f614034b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:45.599 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:45:45.600 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:45:45.600 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:45:45.600 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:45:45.600 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:45:45.601 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:45.601 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:45:45.601 INFO:tasks.workunit.client.0.smithi071.stderr:229903: 0 2021-07-26T06:45:45.601 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:45.601 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:45:45.602 INFO:tasks.workunit.client.0.smithi071.stderr:229903: 2021-07-26T06:45:41.923+0000 7f614034b0c0 -1 Falling back to public interface 2021-07-26T06:45:45.602 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:45.602 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:45.602 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:45.603 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:45:45.603 INFO:tasks.workunit.client.0.smithi071.stderr:229903: 1 2021-07-26T06:45:45.603 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:45.603 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:45:45.604 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:45.604 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:45.604 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:45.604 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:45:45.604 INFO:tasks.workunit.client.0.smithi071.stderr:229903: 2 2021-07-26T06:45:45.605 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:45.605 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:45:45.605 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:45.605 INFO:tasks.workunit.client.0.smithi071.stderr:229903: 2021-07-26T06:45:44.799+0000 7f614034b0c0 -1 osd.1 71 log_to_monitors {default=true} 2021-07-26T06:45:45.606 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:45.606 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:46.851 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:45:46.851 INFO:tasks.workunit.client.0.smithi071.stderr:229905: 3 2021-07-26T06:45:46.852 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:46.852 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:45:46.852 INFO:tasks.workunit.client.0.smithi071.stderr:229905: osd.2 up in weight 1 up_from 76 up_thru 51 down_at 72 last_clean_interval [70,71) [v2:127.0.0.1:6810/230708,v1:127.0.0.1:6811/230708] [v2:127.0.0.1:6812/230708,v1:127.0.0.1:6813/230708] exists,up a1dc4974-84bf-431c-9de1-8a3eb6ab83ae 2021-07-26T06:45:46.852 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:45:46.853 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:45:46.853 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:45:46.853 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:45:46.853 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:45:46.854 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:45:46.854 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:45:46.854 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:45:46.854 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:45:46.855 INFO:tasks.workunit.client.0.smithi071.stderr:229905: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:45:46.855 INFO:tasks.workunit.client.0.smithi071.stderr:229905: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:45:46.855 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:45:46.855 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:45:46.855 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:45:46.856 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:45:46.856 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:45:46.856 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:45:46.856 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:45:46.857 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:45:46.857 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:45:46.857 INFO:tasks.workunit.client.0.smithi071.stderr:229905: 1 2021-07-26T06:45:46.857 INFO:tasks.workunit.client.0.smithi071.stderr:229905: 2 2021-07-26T06:45:46.857 INFO:tasks.workunit.client.0.smithi071.stderr:229905: 3' 2021-07-26T06:45:46.858 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:45:46.858 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:46.858 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:45:46.858 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836522 2021-07-26T06:45:46.859 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836522 2021-07-26T06:45:46.859 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836522' 2021-07-26T06:45:46.859 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:46.859 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:45:46.860 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=326417514498 2021-07-26T06:45:46.860 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 326417514498 2021-07-26T06:45:46.860 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-.sh:925: wait_for_osd: echo 3 2021-07-26T06:45:46.860 INFO:tasks.workunit.client.0.smithi071.stderr:229903: 3 2021-07-26T06:45:46.860 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:46.861 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:45:46.861 INFO:tasks.workunit.client.0.smithi071.stderr:229903: osd.1 up in weight 1 up_from 76 up_thru 70 down_at 72 last_clean_interval [68,71) [v2:127.0.0.1:6818/230733,v1:127.0.0.1:6819/230733] [v2:127.0.0.1:6820/230733,v1:127.0.0.1:6821/230733] exists,up 6a7f7fc8-4b60-487d-84b4-acde9f69054f 2021-07-26T06:45:46.861 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:45:46.861 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:45:46.861 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:45:46.862 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:45:46.862 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:45:46.862 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:45:46.862 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:45:46.863 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:45:46.863 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:45:46.863 INFO:tasks.workunit.client.0.smithi071.stderr:229903: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:45:46.863 INFO:tasks.workunit.client.0.smithi071.stderr:229903: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:45:46.863 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:45:46.864 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:45:46.864 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:45:46.864 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:45:46.864 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:45:46.865 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:45:46.865 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:45:46.865 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:45:46.865 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:45:46.865 INFO:tasks.workunit.client.0.smithi071.stderr:229903: 1 2021-07-26T06:45:46.866 INFO:tasks.workunit.client.0.smithi071.stderr:229903: 2 2021-07-26T06:45:46.866 INFO:tasks.workunit.client.0.smithi071.stderr:229903: 3' 2021-07-26T06:45:46.866 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:45:46.866 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:46.867 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:45:46.867 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836523 2021-07-26T06:45:46.867 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836523 2021-07-26T06:45:46.867 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836523' 2021-07-26T06:45:46.867 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:46.868 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:45:46.868 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=326417514499 2021-07-26T06:45:46.868 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 326417514499 2021-07-26T06:45:49.973 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836523 1-326417514499' 2021-07-26T06:45:49.974 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:49.974 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:45:49.974 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=326417514498 2021-07-26T06:45:49.974 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 326417514498 2021-07-26T06:45:49.975 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836523 1-326417514499 2-326417514498' 2021-07-26T06:45:49.975 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:49.975 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:45:49.975 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=266287972367 2021-07-26T06:45:49.976 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 266287972367 2021-07-26T06:45:49.976 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836523 1-326417514499 2-326417514498 3-266287972367' 2021-07-26T06:45:49.976 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:49.976 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836523 2021-07-26T06:45:49.977 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:49.977 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:45:49.977 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836523 2021-07-26T06:45:49.977 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:49.978 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836523 2021-07-26T06:45:49.978 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2021-07-26T06:45:49.978 INFO:tasks.workunit.client.0.smithi071.stderr:229903: waiting osd.0 seq 21474836523 2021-07-26T06:45:49.978 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:45:49.979 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836521 -lt 21474836523 2021-07-26T06:45:49.979 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:45:49.979 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:45:49.979 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:45:49.980 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836521 -lt 21474836523 2021-07-26T06:45:49.980 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:45:49.980 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:45:49.981 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:45:49.981 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836524 -lt 21474836523 2021-07-26T06:45:49.981 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:49.981 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-326417514499 2021-07-26T06:45:49.982 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:49.982 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:45:49.982 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-326417514499 2021-07-26T06:45:49.983 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //hhelpers.sh:2081: flush_pg_stats: seqs=' 0-21474836522 1-326417514498' 2021-07-26T06:45:49.983 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:49.983 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:45:49.984 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=326417514499 2021-07-26T06:45:49.984 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 326417514499 2021-07-26T06:45:49.984 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836522 1-326417514498 2-326417514499' 2021-07-26T06:45:49.984 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:45:49.985 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:45:49.985 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=266287972366 2021-07-26T06:45:49.985 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 266287972366 2021-07-26T06:45:49.985 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836522 1-326417514498 2-326417514499 3-266287972366' 2021-07-26T06:45:49.985 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:49.986 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836522 2021-07-26T06:45:49.986 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:49.986 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:45:49.986 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836522 2021-07-26T06:45:49.987 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:49.987 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836522 2021-07-26T06:45:49.987 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2021-07-26T06:45:49.987 INFO:tasks.workunit.client.0.smithi071.stderr:229905: waiting osd.0 seq 21474836522 2021-07-26T06:45:49.988 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:45:49.988 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836521 -lt 21474836522 2021-07-26T06:45:49.988 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:45:49.988 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:45:49.988 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:45:49.989 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836521 -lt 21474836522 2021-07-26T06:45:49.989 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:45:49.989 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:45:49.989 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:45:49.990 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836524 -lt 21474836522 2021-07-26T06:45:49.990 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:49.990 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-326417514498 2021-07-26T06:45:49.990 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:49.991 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:45:49.991 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-326417514498 2021-07-26T06:45:51.240 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:51.241 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=326417514499 2021-07-26T06:45:51.242 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 326417514499' 2021-07-26T06:45:51.242 INFO:tasks.workunit.client.0.smithi071.stderr:229903: waiting osd.1 seq 326417514499 2021-07-26T06:45:51.243 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:45:51.243 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 326417514499 -lt 326417514499 2021-07-26T06:45:51.243 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:51.244 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-326417514498 2021-07-26T06:45:51.244 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:51.245 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:45:51.245 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-326417514498 2021-07-26T06:45:51.246 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:51.246 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=326417514498 2021-07-26T06:45:51.247 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 326417514498' 2021-07-26T06:45:51.247 INFO:tasks.workunit.client.0.smithi071.stderr:229903: waiting osd.2 seq 326417514498 2021-07-26T06:45:51.248 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:45:51.248 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 326417514499 -lt 326417514498 2021-07-26T06:45:51.248 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:51.249 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-266287972367 2021-07-26T06:45:51.250 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:51.250 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:45:51.251 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-266287972367 2021-07-26T06:45:51.251 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:51.252 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=266287972367 2021-07-26T06:45:51.252 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 266287972367' 2021-07-26T06:45:51.253 INFO:tasks.workunit.client.0.smithi071.stderr:229903: waiting osd.3 seq 266287972367 2021-07-26T06:45:51.253 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:45:51.253 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 266287972367 -lt 266287972367 2021-07-26T06:45:51.254 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:45:51.254 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:51.255 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:51.255 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:45:51.256 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:51.256 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:51.256 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:51.257 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:51.310 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:51.310 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=326417514498 2021-07-26T06:45:51.311 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 326417514498' 2021-07-26T06:45:51.311 INFO:tasks.workunit.client.0.smithi071.stderr:229905: waiting osd.1 seq 326417514498 2021-07-26T06:45:51.311 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:45:51.312 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 326417514499 -lt 326417514498 2021-07-26T06:45:51.314 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:51.314 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-326417514499 2021-07-26T06:45:51.314 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:51.315 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:45:51.315 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-326417514499 2021-07-26T06:45:51.315 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:51.315 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=326417514499 2021-07-26T06:45:51.315 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 326417514499' 2021-07-26T06:45:51.316 INFO:tasks.workunit.client.0.smithi071.stderr:229905: waiting osd.2 seq 326417514499 2021-07-26T06:45:51.316 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:45:51.316 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 326417514499 -lt 326417514499 2021-07-26T06:45:51.316 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:45:51.317 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-266287972366 2021-07-26T06:45:51.317 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:45:51.317 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:45:51.317 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-266287972366 2021-07-26T06:45:51.317 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:45:51.318 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=266287972366 2021-07-26T06:45:51.318 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 266287972366' 2021-07-26T06:45:51.318 INFO:tasks.workunit.client.0.smithi071.stderr:229905: waiting osd.3 seq 266287972366 2021-07-26T06:45:51.318 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:45:51.318 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 266287972367 -lt 266287972366 2021-07-26T06:45:51.319 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:45:51.319 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:51.319 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:51.319 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:45:51.320 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:51.320 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:51.320 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:51.320 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:51.864 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:51.865 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:51.865 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:51.865 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:45:51.865 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:51.866 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:51.866 INFO:tasks.workunit.client.0.smithi071.stderr:229903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:51.866 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:45:51.866 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:45:51.867 INFO:tasks.workunit.client.0.smithi071.stderr:229903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:45:51.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:45:51.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:45:51.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 229901 2021-07-26T06:45:51.971 INFO:tasks.workunit.client.0.smithi071.stderr:ntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:51.972 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:51.972 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:51.972 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:45:51.973 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:51.973 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:51.973 INFO:tasks.workunit.client.0.smithi071.stderr:229905: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:51.973 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:45:51.974 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:45:51.974 INFO:tasks.workunit.client.0.smithi071.stderr:229905: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:45:51.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:45:51.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:45:51.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:45:51.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:45:51.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2021-07-26T06:45:51.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:45:51.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:45:52.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:45:52.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2021-07-26T06:45:52.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2021-07-26T06:45:52.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2021-07-26T06:45:52.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=3 2021-07-26T06:45:52.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=1 2021-07-26T06:45:52.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2021-07-26T06:45:52.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:45:52.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:45:52.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:45:52.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 232775"' 2021-07-26T06:45:52.007 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 232775' 2021-07-26T06:45:52.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:45:52.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:45:52.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:45:52.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 232776"' 2021-07-26T06:45:52.009 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 232776' 2021-07-26T06:45:52.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2021-07-26T06:45:52.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:45:52.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 232775 232776' 2021-07-26T06:45:52.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:45:52.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:45:52.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 232775 2021-07-26T06:45:52.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/232778: /' 2021-07-26T06:45:52.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:45:52.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/232780: /' 2021-07-26T06:45:54.556 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:45:54.557 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:45:54.557 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:45:54.558 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:45:54.558 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:45:54.559 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:45:54.560 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:45:54.560 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:45:54.560 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:45:54.561 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:45:54.561 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:45:54.561 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:45:54.562 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:45:54.562 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:45:54.563 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:45:54.563 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:45:54.564 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:45:54.564 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:45:54.565 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:45:54.565 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:45:54.565 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:45:54.565 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:45:54.566 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2021-07-26T06:45:54.566 INFO:tasks.workunit.client.0.smithi071.stderr:232780: Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:45:54.566 INFO:tasks.workunit.client.0.smithi071.stderr:232780: Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:45:54.566 INFO:tasks.workunit.client.0.smithi071.stderr:232780: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:45:54.567 INFO:tasks.workunit.client.0.smithi071.stderr:232780: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:45:54.567 INFO:tasks.workunit.client.0.smithi071.stderr:232780: remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:45:54.567 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:45:54.568 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:45:54.568 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:45:54.568 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:45:54.569 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:45:54.570 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:45:54.571 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:45:54.571 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:45:54.571 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:45:54.572 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:45:54.572 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:45:54.573 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:45:54.573 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:45:54.574 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:45:54.574 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:45:54.575 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:45:54.575 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:45:54.576 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:45:54.576 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:45:54.577 INFO:tasks.workunit.client.0.smithi071.stderr:232780: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:45:54.577 INFO:tasks.workunit.client.0.smithi071.stderr:232780: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:45:54.577 INFO:tasks.workunit.client.0.smithi071.stderr:232780: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:45:54.578 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:45:54.578 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:45:54.579 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:45:54.579 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:45:54.579 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:45:54.579 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:45:54.580 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:45:54.580 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:45:54.581 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:45:54.581 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:45:54.581 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:45:54.582 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:45:54.582 INFO:tasks.workunit.client.0.smithi071.stderr:232780: start osd.1 2021-07-26T06:45:54.713 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-di232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:45:54.713 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:45:54.714 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:45:54.714 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:45:54.715 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:45:54.715 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:45:54.715 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:45:54.715 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:45:54.715 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:45:54.716 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:45:54.716 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:45:54.716 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:45:54.716 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:45:54.717 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:45:54.717 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:45:54.717 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:45:54.717 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:45:54.718 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:45:54.718 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:45:54.718 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:45:54.718 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:45:54.718 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:45:54.719 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2021-07-26T06:45:54.719 INFO:tasks.workunit.client.0.smithi071.stderr:232778: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:45:54.719 INFO:tasks.workunit.client.0.smithi071.stderr:232778: Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:45:54.719 INFO:tasks.workunit.client.0.smithi071.stderr:232778: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:45:54.720 INFO:tasks.workunit.client.0.smithi071.stderr:232778: remove 0#2:eb822e21:::SOMETHING:head# 2021-07-26T06:45:54.720 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:45:54.720 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:45:54.720 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:45:54.720 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:45:54.721 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:45:54.721 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:45:54.722 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:45:54.722 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:45:54.722 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:45:54.723 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:45:54.723 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:45:54.723 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:45:54.723 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:45:54.724 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:45:54.724 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:45:54.724 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:45:54.724 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:45:54.724 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:45:54.725 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:45:54.725 INFO:tasks.workunit.client.0.smithi071.stderr:232778: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:45:54.725 INFO:tasks.workunit.client.0.smithi071.stderr:232778: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:45:54.725 INFO:tasks.workunit.client.0.smithi071.stderr:232778: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:45:54.726 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:45:54.726 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:45:54.726 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:45:54.726 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:45:54.726 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:45:54.727 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:45:54.727 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:45:54.727 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:45:54.727 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:45:54.728 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:45:54.728 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:45:54.728 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:45:54.728 INFO:tasks.workunit.client.0.smithi071.stderr:232778: start osd.3 2021-07-26T06:45:58.687 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debr=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:45:58.687 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:45:58.688 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:45:58.688 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:45:58.688 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:45:58.689 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:45:58.690 INFO:tasks.workunit.client.0.smithi071.stderr:232780: 2021-07-26T06:45:54.585+0000 7fa7b921b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:58.690 INFO:tasks.workunit.client.0.smithi071.stderr:232780: 2021-07-26T06:45:54.587+0000 7fa7b921b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:58.690 INFO:tasks.workunit.client.0.smithi071.stderr:232780: 2021-07-26T06:45:54.588+0000 7fa7b921b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:58.690 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:45:58.691 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:45:58.691 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:45:58.691 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:45:58.691 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:45:58.691 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:58.692 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:45:58.692 INFO:tasks.workunit.client.0.smithi071.stderr:232780: 0 2021-07-26T06:45:58.692 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:58.692 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:45:58.693 INFO:tasks.workunit.client.0.smithi071.stderr:232780: 2021-07-26T06:45:55.107+0000 7fa7b921b0c0 -1 Falling back to public interface 2021-07-26T06:45:58.693 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:58.693 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:58.693 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:58.693 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:45:58.694 INFO:tasks.workunit.client.0.smithi071.stderr:232780: 1 2021-07-26T06:45:58.694 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:58.694 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:45:58.694 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:58.695 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:58.695 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:58.695 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:45:58.695 INFO:tasks.workunit.client.0.smithi071.stderr:232780: 2 2021-07-26T06:45:58.696 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:58.696 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:45:58.696 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:58.696 INFO:tasks.workunit.client.0.smithi071.stderr:232780: 2021-07-26T06:45:57.877+0000 7fa7b921b0c0 -1 osd.1 77 log_to_monitors {default=true} 2021-07-26T06:45:58.696 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:58.697 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:58.947 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:45:58.947 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:45:58.948 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:45:58.948 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:45:58.948 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:45:58.948 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:45:58.949 INFO:tasks.workunit.client.0.smithi071.stderr:232778: 2021-07-26T06:45:54.738+0000 7f605848d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:58.949 INFO:tasks.workunit.client.0.smithi071.stderr:232778: 2021-07-26T06:45:54.740+0000 7f605848d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:58.949 INFO:tasks.workunit.client.0.smithi071.stderr:232778: 2021-07-26T06:45:54.741+0000 7f605848d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:45:58.950 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:45:58.950 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:45:58.950 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:45:58.950 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:45:58.950 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:45:58.951 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:58.951 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:45:58.951 INFO:tasks.workunit.client.0.smithi071.stderr:232778: 0 2021-07-26T06:45:58.951 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:58.952 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:45:58.952 INFO:tasks.workunit.client.0.smithi071.stderr:232778: 2021-07-26T06:45:55.256+0000 7f605848d0c0 -1 Falling back to public interface 2021-07-26T06:45:58.952 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:58.952 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:58.952 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:58.953 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:45:58.953 INFO:tasks.workunit.client.0.smithi071.stderr:232778: 1 2021-07-26T06:45:58.953 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:58.953 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:45:58.954 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:58.954 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:58.954 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:58.954 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:45:58.954 INFO:tasks.workunit.client.0.smithi071.stderr:232778: 2 2021-07-26T06:45:58.955 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:45:58.955 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:45:58.955 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:45:58.955 INFO:tasks.workunit.client.0.smithi071.stderr:232778: 2021-07-26T06:45:58.238+0000 7f605848d0c0 -1 osd.3 77 log_to_monitors {default=true} 2021-07-26T06:45:58.956 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:45:58.956 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:45:58.956 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:45:58.957 INFO:tasks.workunit.client.0.smithi071.stderr:232778: 3 2021-07-26T06:46:00.206 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:00.206 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:46:00.206 INFO:tasks.workunit.client.0.smithi071.stderr:232778: osd.3 up in weight 1 up_from 83 up_thru 76 down_at 78 last_clean_interval [62,77) [v2:127.0.0.1:6826/233628,v1:127.0.0.1:6827/233628] [v2:127.0.0.1:6828/233628,v1:127.0.0.1:6829/233628] exists,up b8fbe2eb-a90e-4112-b4de-1f8c84490d2b 2021-07-26T06:46:00.207 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:46:00.207 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:46:00.207 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:46:00.207 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:46:00.208 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:46:00.208 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:46:00.208 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:46:00.208 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:46:00.209 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:46:00.209 INFO:tasks.workunit.client.0.smithi071.stderr:232778: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:46:00.209 INFO:tasks.workunit.client.0.smithi071.stderr:232778: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:46:00.209 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:46:00.209 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:46:00.210 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:46:00.210 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:46:00.210 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:46:00.210 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:46:00.211 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:46:00.211 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:46:00.211 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:46:00.211 INFO:tasks.workunit.client.0.smithi071.stderr:232778: 1 2021-07-26T06:46:00.212 INFO:tasks.workunit.client.0.smithi071.stderr:232778: 2 2021-07-26T06:46:00.212 INFO:tasks.workunit.client.0.smithi071.stderr:232778: 3' 2021-07-26T06:46:00.212 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:46:00.212 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:00.212 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:46:00.213 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836527 2021-07-26T06:46:00.213 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836527 2021-07-26T06:46:00.213 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836527' 2021-07-26T06:46:00.213 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:00.214 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:46:00.214 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=356482285570 2021-07-26T06:46:00.214 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 356482285570 2021-07-26T06:46:00.214 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836527 1-356482285570' 2021-07-26T06:46:00.960 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephait_for_osd: echo 3 2021-07-26T06:46:00.960 INFO:tasks.workunit.client.0.smithi071.stderr:232780: 3 2021-07-26T06:46:00.961 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:00.961 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:46:00.961 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:00.961 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:00.962 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:00.962 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:46:00.962 INFO:tasks.workunit.client.0.smithi071.stderr:232780: 4 2021-07-26T06:46:00.962 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:00.963 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:46:00.963 INFO:tasks.workunit.client.0.smithi071.stderr:232780: osd.1 up in weight 1 up_from 83 up_thru 83 down_at 78 last_clean_interval [76,77) [v2:127.0.0.1:6818/233583,v1:127.0.0.1:6819/233583] [v2:127.0.0.1:6820/233583,v1:127.0.0.1:6821/233583] exists,up 6a7f7fc8-4b60-487d-84b4-acde9f69054f 2021-07-26T06:46:00.963 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:46:00.963 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:46:00.964 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:46:00.964 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:46:00.964 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:46:00.964 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:46:00.965 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:46:00.965 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:46:00.965 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:46:00.965 INFO:tasks.workunit.client.0.smithi071.stderr:232780: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:46:00.965 INFO:tasks.workunit.client.0.smithi071.stderr:232780: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:46:00.966 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:46:00.966 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:46:00.966 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:46:00.966 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:46:00.967 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:46:00.967 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:46:00.967 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:46:00.967 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:46:00.968 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:46:00.968 INFO:tasks.workunit.client.0.smithi071.stderr:232780: 1 2021-07-26T06:46:00.968 INFO:tasks.workunit.client.0.smithi071.stderr:232780: 2 2021-07-26T06:46:00.968 INFO:tasks.workunit.client.0.smithi071.stderr:232780: 3' 2021-07-26T06:46:00.968 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:46:00.969 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:00.969 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:46:00.969 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836528 2021-07-26T06:46:00.969 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836528 2021-07-26T06:46:01.119 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cetest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:01.119 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:46:01.120 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=326417514503 2021-07-26T06:46:01.120 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 326417514503 2021-07-26T06:46:01.120 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836527 1-356482285570 2-326417514503' 2021-07-26T06:46:01.120 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:01.121 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:46:01.121 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=356482285570 2021-07-26T06:46:01.121 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 356482285570 2021-07-26T06:46:01.122 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836527 1-356482285570 2-326417514503 3-356482285570' 2021-07-26T06:46:01.122 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:01.122 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836527 2021-07-26T06:46:01.123 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:01.123 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:46:01.123 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836527 2021-07-26T06:46:01.123 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:01.124 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836527 2021-07-26T06:46:01.124 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2021-07-26T06:46:01.124 INFO:tasks.workunit.client.0.smithi071.stderr:232778: waiting osd.0 seq 21474836527 2021-07-26T06:46:01.124 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:46:01.124 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836527 -lt 21474836527 2021-07-26T06:46:01.125 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:01.125 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-356482285570 2021-07-26T06:46:01.125 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:01.125 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:46:01.126 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-356482285570 2021-07-26T06:46:01.126 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:01.126 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=356482285570 2021-07-26T06:46:01.126 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 356482285570' 2021-07-26T06:46:01.127 INFO:tasks.workunit.client.0.smithi071.stderr:232778: waiting osd.1 seq 356482285570 2021-07-26T06:46:01.127 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:46:01.127 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 356482285570 -lt 356482285570 2021-07-26T06:46:01.127 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:01.127 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-326417514503 2021-07-26T06:46:01.128 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:02.666 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:46:02.666 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-326417514503 2021-07-26T06:46:02.666 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:02.667 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=326417514503 2021-07-26T06:46:02.667 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 326417514503' 2021-07-26T06:46:02.667 INFO:tasks.workunit.client.0.smithi071.stderr:232778: waiting osd.2 seq 326417514503 2021-07-26T06:46:02.667 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:46:02.667 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 326417514503 -lt 326417514503 2021-07-26T06:46:02.668 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:02.668 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-356482285570 2021-07-26T06:46:02.668 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:02.668 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:46:02.669 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-356482285570 2021-07-26T06:46:02.669 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:02.669 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=356482285570 2021-07-26T06:46:02.669 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 356482285570' 2021-07-26T06:46:02.669 INFO:tasks.workunit.client.0.smithi071.stderr:232778: waiting osd.3 seq 356482285570 2021-07-26T06:46:02.670 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:46:02.670 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 356482285570 -lt 356482285570 2021-07-26T06:46:02.670 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:46:02.670 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:02.671 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:02.671 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:46:02.671 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:02.671 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:02.671 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:02.672 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:02.672 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:02.672 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:02.672 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:02.673 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:46:02.673 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:02.673 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:02.673 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:02.954 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_phtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528' 2021-07-26T06:46:02.954 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:02.954 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:46:02.955 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=356482285571 2021-07-26T06:46:02.955 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 356482285571 2021-07-26T06:46:02.955 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528 1-356482285571' 2021-07-26T06:46:02.955 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:02.955 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:46:02.956 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=326417514504 2021-07-26T06:46:02.956 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 326417514504 2021-07-26T06:46:02.956 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528 1-356482285571 2-326417514504' 2021-07-26T06:46:02.956 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:02.957 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:46:02.957 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=356482285571 2021-07-26T06:46:02.957 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 356482285571 2021-07-26T06:46:02.957 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528 1-356482285571 2-326417514504 3-356482285571' 2021-07-26T06:46:02.957 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:02.958 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836528 2021-07-26T06:46:02.958 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:02.958 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:46:02.958 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836528 2021-07-26T06:46:02.959 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:02.959 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836528 2021-07-26T06:46:02.959 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2021-07-26T06:46:02.959 INFO:tasks.workunit.client.0.smithi071.stderr:232780: waiting osd.0 seq 21474836528 2021-07-26T06:46:02.959 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:46:02.960 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836527 -lt 21474836528 2021-07-26T06:46:02.960 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:46:02.960 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:46:02.960 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:46:02.960 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836529 -lt 21474836528 2021-07-26T06:46:02.961 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:02.961 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-356482285571 2021-07-26T06:46:02.961 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:03.404 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.for_clean: test 4 = 5 2021-07-26T06:46:03.404 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 4 '!=' -1 2021-07-26T06:46:03.404 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:46:03.405 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=4 2021-07-26T06:46:03.405 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:03.405 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:03.405 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:03.405 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:03.406 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:03.406 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:03.406 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:03.406 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:03.407 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:03.407 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:03.407 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:46:03.407 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:03.408 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:03.408 INFO:tasks.workunit.client.0.smithi071.stderr:232778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:03.408 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:46:03.408 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:46:03.408 INFO:tasks.workunit.client.0.smithi071.stderr:232778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:46:03.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:46:03.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:46:03.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 232776 2021-07-26T06:46:04.191 INFO:tasks.workunit.client.0.smithi071.stderr:sh:2085: flush_pg_stats: osd=1 2021-07-26T06:46:04.192 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-356482285571 2021-07-26T06:46:04.192 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:04.192 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=356482285571 2021-07-26T06:46:04.192 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 356482285571' 2021-07-26T06:46:04.193 INFO:tasks.workunit.client.0.smithi071.stderr:232780: waiting osd.1 seq 356482285571 2021-07-26T06:46:04.193 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:46:04.193 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 356482285571 -lt 356482285571 2021-07-26T06:46:04.193 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:04.193 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-326417514504 2021-07-26T06:46:04.194 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:04.194 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:46:04.194 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-326417514504 2021-07-26T06:46:04.194 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:04.195 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=326417514504 2021-07-26T06:46:04.195 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 326417514504' 2021-07-26T06:46:04.195 INFO:tasks.workunit.client.0.smithi071.stderr:232780: waiting osd.2 seq 326417514504 2021-07-26T06:46:04.195 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:46:04.196 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 326417514504 -lt 326417514504 2021-07-26T06:46:04.196 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:04.196 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-356482285571 2021-07-26T06:46:04.196 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:04.196 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:46:04.197 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-356482285571 2021-07-26T06:46:04.197 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:04.197 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=356482285571 2021-07-26T06:46:04.197 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 356482285571' 2021-07-26T06:46:04.197 INFO:tasks.workunit.client.0.smithi071.stderr:232780: waiting osd.3 seq 356482285571 2021-07-26T06:46:04.198 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:46:04.198 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 356482285571 -lt 356482285571 2021-07-26T06:46:04.198 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:46:04.198 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:04.199 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:04.199 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:46:04.199 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:04.199 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:04.200 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:04.779 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:04.779 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:04.779 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:04.779 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:04.780 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:46:04.780 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:04.780 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:04.780 INFO:tasks.workunit.client.0.smithi071.stderr:232780: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:04.781 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:46:04.781 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:46:04.781 INFO:tasks.workunit.client.0.smithi071.stderr:232780: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:46:04.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:46:04.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:46:04.782 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:46:04.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:46:04.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2021-07-26T06:46:04.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:46:04.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2021-07-26T06:46:04.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:46:04.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:46:04.784 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:46:04.784 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:46:05.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2021-07-26T06:46:05.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2021-07-26T06:46:05.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:46:05.049 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:46:05.049 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:46:05.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:46:05.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:46:05.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:46:05.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:45:32.421576+0000 2021-07-26T06:46:05.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:46:05.566 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:46:05.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:45:32.421576+0000 2021-07-26T06:46:05.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:46:05.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:45:32.421576+0000 2021-07-26T06:46:05.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:46:05.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:46:05.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:46:05.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:46:05.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:46:05.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:46:05.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:46:05.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:46:05.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:45:32.421576+0000 '>' 2021-07-26T06:45:32.421576+0000 2021-07-26T06:46:05.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:46:06.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:46:06.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:46:06.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:46:06.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:46:06.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:46:06.841 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:46:06.841 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:46:07.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:45:32.421576+0000 '>' 2021-07-26T06:45:32.421576+0000 2021-07-26T06:46:07.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:46:08.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:46:08.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:46:08.114 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:46:08.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:46:08.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:46:08.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:46:08.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:46:08.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:45:32.421576+0000 '>' 2021-07-26T06:45:32.421576+0000 2021-07-26T06:46:08.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:46:09.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:46:09.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:46:09.398 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:46:09.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:46:09.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:46:09.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:46:09.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:46:09.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:45:32.421576+0000 '>' 2021-07-26T06:45:32.421576+0000 2021-07-26T06:46:09.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:46:10.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:46:10.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:46:10.672 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:46:10.673 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:46:10.673 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:46:10.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:46:10.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:46:10.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:46:06.391038+0000 '>' 2021-07-26T06:45:32.421576+0000 2021-07-26T06:46:10.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:46:10.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2021-07-26T06:46:10.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:46:10.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:46:10.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:46:10.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 235835"' 2021-07-26T06:46:10.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 235835' 2021-07-26T06:46:10.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:46:10.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:46:10.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:46:10.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 235836"' 2021-07-26T06:46:10.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 235836' 2021-07-26T06:46:10.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2021-07-26T06:46:10.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 235835 235836' 2021-07-26T06:46:10.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:46:10.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:46:10.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 235835 2021-07-26T06:46:10.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:46:10.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/235838: /' 2021-07-26T06:46:10.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:46:10.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/235840: /' 2021-07-26T06:46:13.318 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:46:13.318 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:46:13.319 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:46:13.319 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:46:13.319 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:46:13.319 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:46:13.320 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:46:13.320 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:46:13.320 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:46:13.320 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:46:13.321 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:46:13.321 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:46:13.321 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:46:13.321 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:46:13.322 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:46:13.322 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:46:13.322 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:46:13.322 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:46:13.323 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:46:13.323 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:46:13.323 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:46:13.323 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:46:13.324 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2021-07-26T06:46:13.324 INFO:tasks.workunit.client.0.smithi071.stderr:235838: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:46:13.324 INFO:tasks.workunit.client.0.smithi071.stderr:235838: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2021-07-26T06:46:13.324 INFO:tasks.workunit.client.0.smithi071.stderr:235838: Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:46:13.325 INFO:tasks.workunit.client.0.smithi071.stderr:235838: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:46:13.325 INFO:tasks.workunit.client.0.smithi071.stderr:235838: _ 2021-07-26T06:46:13.325 INFO:tasks.workunit.client.0.smithi071.stderr:235838: hinfo_key 2021-07-26T06:46:13.325 INFO:tasks.workunit.client.0.smithi071.stderr:235838: snapset 2021-07-26T06:46:13.326 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:46:13.326 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:46:13.326 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:46:13.326 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:46:13.327 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:46:13.327 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:46:13.327 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:46:13.328 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:46:13.328 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:46:13.328 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:46:13.328 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:46:13.329 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:46:13.329 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:46:13.329 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:46:13.329 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:46:13.330 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:46:13.330 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:46:13.330 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:46:13.330 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:46:13.330 INFO:tasks.workunit.client.0.smithi071.stderr:235838: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:46:13.331 INFO:tasks.workunit.client.0.smithi071.stderr:235838: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:13.331 INFO:tasks.workunit.client.0.smithi071.stderr:235838: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:13.331 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:13.331 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:13.332 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:46:13.332 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:46:13.332 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:46:13.332 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:46:13.333 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:46:13.333 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:13.333 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:46:13.333 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:46:13.333 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:46:13.334 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:46:13.334 INFO:tasks.workunit.client.0.smithi071.stderr:235838: start osd.3 2021-07-26T06:46:13.541 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-qu235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:46:13.541 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:46:13.542 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:46:13.542 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:46:13.542 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:46:13.542 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:46:13.543 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:46:13.543 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:46:13.543 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:46:13.543 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:46:13.544 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:46:13.544 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:46:13.544 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:46:13.545 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:46:13.545 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:46:13.545 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:46:13.545 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:46:13.546 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:46:13.546 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:46:13.546 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:46:13.546 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:46:13.547 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:46:13.547 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2021-07-26T06:46:13.547 INFO:tasks.workunit.client.0.smithi071.stderr:235840: Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:46:13.547 INFO:tasks.workunit.client.0.smithi071.stderr:235840: Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:46:13.548 INFO:tasks.workunit.client.0.smithi071.stderr:235840: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:46:13.548 INFO:tasks.workunit.client.0.smithi071.stderr:235840: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:46:13.548 INFO:tasks.workunit.client.0.smithi071.stderr:235840: _ 2021-07-26T06:46:13.548 INFO:tasks.workunit.client.0.smithi071.stderr:235840: hinfo_key 2021-07-26T06:46:13.549 INFO:tasks.workunit.client.0.smithi071.stderr:235840: snapset 2021-07-26T06:46:13.549 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:46:13.549 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:46:13.549 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:46:13.550 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:46:13.550 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:46:13.550 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:46:13.551 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:46:13.551 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:46:13.551 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:46:13.551 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:46:13.552 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:46:13.552 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:46:13.552 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:46:13.552 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:46:13.553 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:46:13.553 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:46:13.553 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:46:13.553 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:46:13.553 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:46:13.554 INFO:tasks.workunit.client.0.smithi071.stderr:235840: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:46:13.554 INFO:tasks.workunit.client.0.smithi071.stderr:235840: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:13.554 INFO:tasks.workunit.client.0.smithi071.stderr:235840: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:13.555 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:13.555 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:13.555 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:46:13.556 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:46:13.556 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:46:13.556 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:46:13.556 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:46:13.557 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:13.557 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:46:13.557 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:46:13.557 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:46:13.558 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:46:13.558 INFO:tasks.workunit.client.0.smithi071.stderr:235840: start osd.1 2021-07-26T06:46:17.479 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdireue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:17.479 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:46:17.479 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:46:17.480 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:46:17.480 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:46:17.480 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:46:17.481 INFO:tasks.workunit.client.0.smithi071.stderr:235838: 2021-07-26T06:46:13.345+0000 7f921c49e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:17.481 INFO:tasks.workunit.client.0.smithi071.stderr:235838: 2021-07-26T06:46:13.348+0000 7f921c49e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:17.481 INFO:tasks.workunit.client.0.smithi071.stderr:235838: 2021-07-26T06:46:13.349+0000 7f921c49e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:17.481 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:46:17.481 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:46:17.482 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:46:17.482 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:46:17.482 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:46:17.482 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:17.483 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:46:17.483 INFO:tasks.workunit.client.0.smithi071.stderr:235838: 0 2021-07-26T06:46:17.483 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:17.483 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:46:17.484 INFO:tasks.workunit.client.0.smithi071.stderr:235838: 2021-07-26T06:46:13.867+0000 7f921c49e0c0 -1 Falling back to public interface 2021-07-26T06:46:17.484 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:17.484 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:17.484 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:17.485 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:46:17.485 INFO:tasks.workunit.client.0.smithi071.stderr:235838: 1 2021-07-26T06:46:17.485 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:17.485 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:46:17.485 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:17.486 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:17.486 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:17.486 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:46:17.486 INFO:tasks.workunit.client.0.smithi071.stderr:235838: 2 2021-07-26T06:46:17.487 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:17.487 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:46:17.487 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:17.488 INFO:tasks.workunit.client.0.smithi071.stderr:235838: 2021-07-26T06:46:16.485+0000 7f921c49e0c0 -1 osd.3 85 log_to_monitors {default=true} 2021-07-26T06:46:17.488 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:17.488 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:17.748 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalo= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:17.749 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:46:17.749 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:46:17.749 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:46:17.749 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:46:17.750 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:46:17.750 INFO:tasks.workunit.client.0.smithi071.stderr:235840: 2021-07-26T06:46:13.566+0000 7f6f31d7d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:17.750 INFO:tasks.workunit.client.0.smithi071.stderr:235840: 2021-07-26T06:46:13.568+0000 7f6f31d7d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:17.750 INFO:tasks.workunit.client.0.smithi071.stderr:235840: 2021-07-26T06:46:13.569+0000 7f6f31d7d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:17.750 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:46:17.751 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:46:17.751 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:46:17.751 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:46:17.751 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:46:17.752 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:17.752 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:46:17.752 INFO:tasks.workunit.client.0.smithi071.stderr:235840: 0 2021-07-26T06:46:17.752 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:17.752 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:46:17.753 INFO:tasks.workunit.client.0.smithi071.stderr:235840: 2021-07-26T06:46:14.092+0000 7f6f31d7d0c0 -1 Falling back to public interface 2021-07-26T06:46:17.753 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:17.753 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:17.753 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:17.754 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:46:17.754 INFO:tasks.workunit.client.0.smithi071.stderr:235840: 1 2021-07-26T06:46:17.754 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:17.754 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:46:17.754 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:17.755 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:17.755 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:17.755 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:46:17.755 INFO:tasks.workunit.client.0.smithi071.stderr:235840: 2 2021-07-26T06:46:17.756 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:17.756 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:46:17.756 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:17.756 INFO:tasks.workunit.client.0.smithi071.stderr:235840: 2021-07-26T06:46:16.913+0000 7f6f31d7d0c0 -1 osd.1 85 log_to_monitors {default=true} 2021-07-26T06:46:17.756 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:17.757 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:18.627 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpersne/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:46:18.627 INFO:tasks.workunit.client.0.smithi071.stderr:235838: 3 2021-07-26T06:46:18.627 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:18.628 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:46:18.628 INFO:tasks.workunit.client.0.smithi071.stderr:235838: osd.3 up in weight 1 up_from 90 up_thru 90 down_at 86 last_clean_interval [83,85) [v2:127.0.0.1:6818/236643,v1:127.0.0.1:6819/236643] [v2:127.0.0.1:6820/236643,v1:127.0.0.1:6821/236643] exists,up b8fbe2eb-a90e-4112-b4de-1f8c84490d2b 2021-07-26T06:46:18.628 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:46:18.629 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:46:18.629 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:46:18.629 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:46:18.630 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:46:18.630 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:46:18.630 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:46:18.631 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:46:18.631 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:46:18.631 INFO:tasks.workunit.client.0.smithi071.stderr:235838: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:46:18.631 INFO:tasks.workunit.client.0.smithi071.stderr:235838: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:46:18.631 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:46:18.632 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:46:18.632 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:46:18.632 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:46:18.632 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:46:18.633 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:46:18.633 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:46:18.633 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:46:18.633 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:46:18.634 INFO:tasks.workunit.client.0.smithi071.stderr:235838: 1 2021-07-26T06:46:18.634 INFO:tasks.workunit.client.0.smithi071.stderr:235838: 2 2021-07-26T06:46:18.634 INFO:tasks.workunit.client.0.smithi071.stderr:235838: 3' 2021-07-26T06:46:18.634 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:46:18.635 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:18.635 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:46:18.635 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836533 2021-07-26T06:46:18.635 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836533 2021-07-26T06:46:18.635 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836533' 2021-07-26T06:46:18.636 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:18.636 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:46:18.636 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=390842023938 2021-07-26T06:46:18.636 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 390842023938 2021-07-26T06:46:19.003 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/s.sh:925: wait_for_osd: echo 3 2021-07-26T06:46:19.004 INFO:tasks.workunit.client.0.smithi071.stderr:235840: 3 2021-07-26T06:46:19.004 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:19.005 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:46:19.005 INFO:tasks.workunit.client.0.smithi071.stderr:235840: osd.1 up in weight 1 up_from 91 up_thru 83 down_at 86 last_clean_interval [83,85) [v2:127.0.0.1:6826/236688,v1:127.0.0.1:6827/236688] [v2:127.0.0.1:6828/236688,v1:127.0.0.1:6829/236688] exists,up 6a7f7fc8-4b60-487d-84b4-acde9f69054f 2021-07-26T06:46:19.006 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:46:19.006 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:46:19.007 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:46:19.007 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:46:19.007 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:46:19.008 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:46:19.008 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:46:19.009 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:46:19.009 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:46:19.010 INFO:tasks.workunit.client.0.smithi071.stderr:235840: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:46:19.010 INFO:tasks.workunit.client.0.smithi071.stderr:235840: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:46:19.011 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:46:19.011 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:46:19.011 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:46:19.012 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:46:19.012 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:46:19.013 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:46:19.013 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:46:19.013 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:46:19.014 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:46:19.014 INFO:tasks.workunit.client.0.smithi071.stderr:235840: 1 2021-07-26T06:46:19.015 INFO:tasks.workunit.client.0.smithi071.stderr:235840: 2 2021-07-26T06:46:19.015 INFO:tasks.workunit.client.0.smithi071.stderr:235840: 3' 2021-07-26T06:46:19.015 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:46:19.016 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:19.016 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:46:19.017 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836534 2021-07-26T06:46:19.017 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836534 2021-07-26T06:46:19.018 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534' 2021-07-26T06:46:19.018 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:19.019 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:46:19.019 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=390842023939 2021-07-26T06:46:19.020 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 390842023939 2021-07-26T06:46:19.581 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-tandalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836533 1-390842023938' 2021-07-26T06:46:19.582 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:19.582 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:46:19.582 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=326417514508 2021-07-26T06:46:19.582 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 326417514508 2021-07-26T06:46:19.583 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836533 1-390842023938 2-326417514508' 2021-07-26T06:46:19.583 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:19.583 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:46:19.583 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=386547056642 2021-07-26T06:46:19.584 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 386547056642 2021-07-26T06:46:19.584 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836533 1-390842023938 2-326417514508 3-386547056642' 2021-07-26T06:46:19.584 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:19.584 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836533 2021-07-26T06:46:19.585 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:19.585 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:46:19.585 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836533 2021-07-26T06:46:19.585 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:19.585 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836533 2021-07-26T06:46:19.586 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2021-07-26T06:46:19.586 INFO:tasks.workunit.client.0.smithi071.stderr:235838: waiting osd.0 seq 21474836533 2021-07-26T06:46:19.586 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:46:19.587 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836533 -lt 21474836533 2021-07-26T06:46:19.587 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:19.587 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-390842023938 2021-07-26T06:46:19.587 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:19.588 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:46:19.588 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-390842023938 2021-07-26T06:46:19.588 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:19.588 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=390842023938 2021-07-26T06:46:19.588 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 390842023938' 2021-07-26T06:46:19.589 INFO:tasks.workunit.client.0.smithi071.stderr:235838: waiting osd.1 seq 390842023938 2021-07-26T06:46:19.589 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:46:19.589 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 390842023938 -lt 390842023938 2021-07-26T06:46:19.589 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:19.590 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-326417514508 2021-07-26T06:46:20.856 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalonhelpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534 1-390842023939' 2021-07-26T06:46:20.856 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:20.856 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:46:20.857 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=326417514509 2021-07-26T06:46:20.857 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 326417514509 2021-07-26T06:46:20.857 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534 1-390842023939 2-326417514509' 2021-07-26T06:46:20.857 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:46:20.857 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:46:20.858 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=386547056643 2021-07-26T06:46:20.858 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 386547056643 2021-07-26T06:46:20.858 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534 1-390842023939 2-326417514509 3-386547056643' 2021-07-26T06:46:20.858 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:20.859 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836534 2021-07-26T06:46:20.859 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:20.859 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:46:20.859 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836534 2021-07-26T06:46:20.860 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:20.860 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836534 2021-07-26T06:46:20.860 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2021-07-26T06:46:20.860 INFO:tasks.workunit.client.0.smithi071.stderr:235840: waiting osd.0 seq 21474836534 2021-07-26T06:46:20.860 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:46:20.861 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836533 -lt 21474836534 2021-07-26T06:46:20.861 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:46:20.861 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:46:20.861 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:46:20.861 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836534 -lt 21474836534 2021-07-26T06:46:20.862 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:20.862 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-390842023939 2021-07-26T06:46:20.862 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:20.862 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:46:20.863 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-390842023939 2021-07-26T06:46:20.863 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:20.863 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=390842023939 2021-07-26T06:46:20.863 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 390842023939' 2021-07-26T06:46:20.863 INFO:tasks.workunit.client.0.smithi071.stderr:235840: waiting osd.1 seq 390842023939 2021-07-26T06:46:21.875 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flue/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:21.875 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:46:21.875 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-326417514508 2021-07-26T06:46:21.876 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:21.876 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=326417514508 2021-07-26T06:46:21.876 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 326417514508' 2021-07-26T06:46:21.876 INFO:tasks.workunit.client.0.smithi071.stderr:235838: waiting osd.2 seq 326417514508 2021-07-26T06:46:21.877 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:46:21.877 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 326417514507 -lt 326417514508 2021-07-26T06:46:21.877 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:46:21.877 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:46:21.878 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:46:21.878 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 326417514510 -lt 326417514508 2021-07-26T06:46:21.878 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:21.878 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-386547056642 2021-07-26T06:46:21.878 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:21.879 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:46:21.879 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-386547056642 2021-07-26T06:46:21.879 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:21.879 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=386547056642 2021-07-26T06:46:21.880 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 386547056642' 2021-07-26T06:46:21.880 INFO:tasks.workunit.client.0.smithi071.stderr:235838: waiting osd.3 seq 386547056642 2021-07-26T06:46:21.880 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:46:21.880 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 386547056643 -lt 386547056642 2021-07-26T06:46:21.880 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:46:21.881 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:21.881 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:21.881 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:46:21.881 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:21.881 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:21.882 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:21.882 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:21.882 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:21.882 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:22.120 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | selsh_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:46:22.120 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 390842023939 -lt 390842023939 2021-07-26T06:46:22.121 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:22.121 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-326417514509 2021-07-26T06:46:22.121 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:22.121 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:46:22.122 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-326417514509 2021-07-26T06:46:22.122 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:22.122 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=326417514509 2021-07-26T06:46:22.128 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 326417514509' 2021-07-26T06:46:22.128 INFO:tasks.workunit.client.0.smithi071.stderr:235840: waiting osd.2 seq 326417514509 2021-07-26T06:46:22.129 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:46:22.129 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 326417514510 -lt 326417514509 2021-07-26T06:46:22.129 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:46:22.129 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-386547056643 2021-07-26T06:46:22.130 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:46:22.130 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:46:22.130 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-386547056643 2021-07-26T06:46:22.130 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:46:22.131 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=386547056643 2021-07-26T06:46:22.131 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 386547056643' 2021-07-26T06:46:22.131 INFO:tasks.workunit.client.0.smithi071.stderr:235840: waiting osd.3 seq 386547056643 2021-07-26T06:46:22.131 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:46:22.131 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 386547056643 -lt 386547056643 2021-07-26T06:46:22.132 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:46:22.132 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:22.132 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:22.132 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:46:22.133 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:22.133 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:22.133 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:22.133 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:22.133 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:22.134 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:22.508 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("staect(contains("stale") | not)] | length' 2021-07-26T06:46:22.508 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:46:22.508 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:22.508 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:22.509 INFO:tasks.workunit.client.0.smithi071.stderr:235838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:22.509 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:46:22.509 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:46:22.509 INFO:tasks.workunit.client.0.smithi071.stderr:235838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:46:22.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:46:22.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:46:22.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 235836 2021-07-26T06:46:22.751 INFO:tasks.workunit.client.0.smithi071.stderr:le") | not)] | length' 2021-07-26T06:46:22.751 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:46:22.751 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:22.751 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:22.752 INFO:tasks.workunit.client.0.smithi071.stderr:235840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:22.752 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:46:22.752 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:46:22.752 INFO:tasks.workunit.client.0.smithi071.stderr:235840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:46:22.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:46:22.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:46:22.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:46:22.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:46:22.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2021-07-26T06:46:22.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:46:22.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:46:22.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:46:22.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:46:22.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:46:22.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:46:22.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:46:22.790 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:46:22.790 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:46:22.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:46:22.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:46:22.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:46:22.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:46:22.898 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:46:22.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:46:22.899 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:46:22.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:46:22.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:46:22.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:46:22.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:46:22.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:46:22.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:46:22.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:46:22.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:46:22.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:46:22.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:46:22.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:46:22.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:22.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:22.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:46:22.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:46:22.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:46:22.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:46:22.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:46:22.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:46:22.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:46:22.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:46:22.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:46:22.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:46:22.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:46:22.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:46:22.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:46:22.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:46:22.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:46:22.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:46:22.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:46:22.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:46:22.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:46:22.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:46:22.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:46:22.932 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:46:22.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:46:22.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:46:22.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:46:22.933 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:46:22.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:46:22.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:46:22.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:46:22.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:46:22.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:22.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:22.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:46:22.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:46:22.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:46:22.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:46:22.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:46:22.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:22.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:22.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:46:22.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:46:22.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:46:22.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:46:22.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:46:22.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2021-07-26T06:46:22.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2021-07-26T06:46:22.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2021-07-26T06:46:22.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local allow_overwrites=false 2021-07-26T06:46:22.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:828: corrupt_and_repair_lrc: local poolname=ecpool 2021-07-26T06:46:22.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2021-07-26T06:46:22.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:46:22.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:46:22.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:46:22.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:46:22.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:46:22.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T06:46:23.295 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:46:23.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:46:23.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:46:23.443 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:46:23.443 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:23.444 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:23.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:23.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:46:23.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:46:23.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:46:23.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:46:23.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:46:23.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:46:23.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:46:23.623 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:46:23.623 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:46:23.623 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:46:23.623 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:46:23.624 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:23.624 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:23.624 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:46:23.625 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:46:23.625 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:46:23.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:46:23.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:46:23.720 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:46:23.720 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:46:23.720 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:46:23.720 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:46:23.720 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:46:23.721 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:46:23.721 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:46:23.721 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:23.721 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:23.722 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:46:23.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:46:23.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:46:23.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2021-07-26T06:46:23.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:46:23.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:46:23.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:46:23.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:46:23.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:46:23.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:46:24.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:46:24.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:46:24.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:46:24.651 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:46:24.651 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:24.652 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:24.652 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:24.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:46:24.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:46:24.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: seq 0 9 2021-07-26T06:46:24.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:46:24.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' false = true ']' 2021-07-26T06:46:24.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 0 2021-07-26T06:46:24.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:46:24.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:46:24.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2021-07-26T06:46:24.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:46:24.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:46:24.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:46:24.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:46:24.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:46:24.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:46:24.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:46:24.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:46:24.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:46:24.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:46:24.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:46:24.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:46:24.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:46:24.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:46:24.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:46:24.683 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:46:24.684 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:24.684 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:24.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:24.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:24.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:46:24.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:46:24.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:46:24.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:46:24.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:46:24.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:46:24.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:46:24.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:24.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:46:24.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:46:24.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:46:24.687 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:46:24.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=f24877a1-c60a-45ed-b04f-6a9687cef326 2021-07-26T06:46:24.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd3 f24877a1-c60a-45ed-b04f-6a9687cef326' 2021-07-26T06:46:24.688 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 f24877a1-c60a-45ed-b04f-6a9687cef326 2021-07-26T06:46:24.688 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:46:24.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQBAWv5gb0dhKRAAH/uRS28PDZxH232x0hQzqA== 2021-07-26T06:46:24.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQBAWv5gb0dhKRAAH/uRS28PDZxH232x0hQzqA=="}' 2021-07-26T06:46:24.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new f24877a1-c60a-45ed-b04f-6a9687cef326 -i td/osd-scrub-repair/0/new.json 2021-07-26T06:46:24.948 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:46:24.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:46:24.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBAWv5gb0dhKRAAH/uRS28PDZxH232x0hQzqA== --osd-uuid f24877a1-c60a-45ed-b04f-6a9687cef326 --osd-objectstore=filestore 2021-07-26T06:46:24.984 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:24.985+0000 7f0492a090c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:24.986 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:24.987+0000 7f0492a090c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:24.987 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:24.988+0000 7f0492a090c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:25.468 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:25.469+0000 7f0492a090c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:25.905 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:25.906+0000 7f0492a090c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:25.905 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:25.906+0000 7f0492a090c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:25.906 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:25.906+0000 7f0492a090c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:26.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:46:26.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:46:26.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd0 key to auth repository 2021-07-26T06:46:26.399 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:46:26.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:46:26.571 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2021-07-26T06:46:26.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.0 2021-07-26T06:46:26.573 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:46:26.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:26.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:46:26.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:46:26.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:46:26.596 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:26.597+0000 7f9bf93b40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:26.598 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:26.599+0000 7f9bf93b40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:26.599 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:26.600+0000 7f9bf93b40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:26.600 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:26.601+0000 7f9bf93b40c0 -1 Falling back to public interface 2021-07-26T06:46:26.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 0 2021-07-26T06:46:26.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:46:26.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:46:26.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:46:26.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:46:26.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:26.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:46:26.856 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:46:26.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:26.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:46:27.029 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:27.029+0000 7f9bf93b40c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:27.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:27.030 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:27.030+0000 7f9bf93b40c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:27.030 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:27.030+0000 7f9bf93b40c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:27.036 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:27.037+0000 7f9bf93b40c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:46:28.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:28.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:28.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:46:28.031 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:46:28.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:28.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:46:28.224 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/239021,v1:127.0.0.1:6801/239021] [v2:127.0.0.1:6802/239021,v1:127.0.0.1:6803/239021] exists,up f24877a1-c60a-45ed-b04f-6a9687cef326 2021-07-26T06:46:28.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:46:28.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:46:28.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:46:28.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:46:28.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' false = true ']' 2021-07-26T06:46:28.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 1 2021-07-26T06:46:28.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:46:28.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:46:28.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2021-07-26T06:46:28.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:46:28.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:46:28.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:46:28.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:46:28.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:46:28.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:46:28.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:46:28.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:46:28.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:46:28.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:46:28.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:46:28.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:46:28.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:46:28.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:46:28.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:46:28.235 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:46:28.235 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:28.236 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:28.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:28.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:28.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:46:28.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:46:28.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:46:28.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:46:28.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:46:28.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:46:28.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:46:28.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:28.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:46:28.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:46:28.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:46:28.239 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:46:28.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=17242162-1c8e-49a2-8d6f-f74428030f4f 2021-07-26T06:46:28.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd3 17242162-1c8e-49a2-8d6f-f74428030f4f' 2021-07-26T06:46:28.239 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 17242162-1c8e-49a2-8d6f-f74428030f4f 2021-07-26T06:46:28.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:46:28.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQBEWv5g27WBDhAAEZOwAJVZiiuP0VcjfuBjZA== 2021-07-26T06:46:28.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQBEWv5g27WBDhAAEZOwAJVZiiuP0VcjfuBjZA=="}' 2021-07-26T06:46:28.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 17242162-1c8e-49a2-8d6f-f74428030f4f -i td/osd-scrub-repair/1/new.json 2021-07-26T06:46:28.518 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:46:28.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:46:28.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBEWv5g27WBDhAAEZOwAJVZiiuP0VcjfuBjZA== --osd-uuid 17242162-1c8e-49a2-8d6f-f74428030f4f --osd-objectstore=filestore 2021-07-26T06:46:28.554 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:28.554+0000 7f63e5b9a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:28.568 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:28.568+0000 7f63e5b9a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:28.569 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:28.569+0000 7f63e5b9a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:29.139 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:29.139+0000 7f63e5b9a0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:29.828 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:29.828+0000 7f63e5b9a0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:29.836 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:29.836+0000 7f63e5b9a0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:29.836 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:29.836+0000 7f63e5b9a0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:30.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:46:30.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:46:30.601 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:46:30.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd1 key to auth repository 2021-07-26T06:46:30.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:46:30.931 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2021-07-26T06:46:30.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.1 2021-07-26T06:46:30.936 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:46:30.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:30.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:46:30.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:46:30.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:46:30.955 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:30.955+0000 7f475d8800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:30.956 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:30.957+0000 7f475d8800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:30.957 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:30.957+0000 7f475d8800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:30.958 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:30.959+0000 7f475d8800c0 -1 Falling back to public interface 2021-07-26T06:46:31.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 1 2021-07-26T06:46:31.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:46:31.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:46:31.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:46:31.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:46:31.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:31.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:46:31.366 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:46:31.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:31.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:46:31.462 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:31.462+0000 7f475d8800c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:31.475 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:31.475+0000 7f475d8800c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:31.475 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:31.475+0000 7f475d8800c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:31.480 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:31.481+0000 7f475d8800c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:46:31.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:32.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:32.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:32.694 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:46:32.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:46:32.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:32.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:46:32.967 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/239393,v1:127.0.0.1:6811/239393] [v2:127.0.0.1:6812/239393,v1:127.0.0.1:6813/239393] exists,up 17242162-1c8e-49a2-8d6f-f74428030f4f 2021-07-26T06:46:32.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:46:32.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:46:32.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:46:32.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:46:32.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' false = true ']' 2021-07-26T06:46:32.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 2 2021-07-26T06:46:32.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:46:32.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:46:32.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2021-07-26T06:46:32.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:46:32.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:46:32.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:46:32.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:46:32.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:46:32.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:46:32.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:46:32.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:46:32.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:46:32.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:46:32.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:46:32.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:46:32.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:46:32.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:46:32.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:46:32.978 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:46:32.978 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:32.978 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:32.979 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:32.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:32.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:46:32.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:46:32.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:46:32.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:46:32.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:46:32.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:46:32.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:46:32.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:32.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:46:32.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:46:32.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:46:32.982 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:46:32.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=ea4746a6-9c33-403d-9449-debe3bf0045e 2021-07-26T06:46:32.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd3 ea4746a6-9c33-403d-9449-debe3bf0045e' 2021-07-26T06:46:32.982 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 ea4746a6-9c33-403d-9449-debe3bf0045e 2021-07-26T06:46:32.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:46:32.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQBIWv5gxTHKOhAAJSzFRevB33npjoNdN9V5dQ== 2021-07-26T06:46:32.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQBIWv5gxTHKOhAAJSzFRevB33npjoNdN9V5dQ=="}' 2021-07-26T06:46:32.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new ea4746a6-9c33-403d-9449-debe3bf0045e -i td/osd-scrub-repair/2/new.json 2021-07-26T06:46:33.373 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:46:33.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2021-07-26T06:46:33.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBIWv5gxTHKOhAAJSzFRevB33npjoNdN9V5dQ== --osd-uuid ea4746a6-9c33-403d-9449-debe3bf0045e --osd-objectstore=filestore 2021-07-26T06:46:33.410 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:33.410+0000 7f11191290c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:33.414 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:33.414+0000 7f11191290c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:33.415 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:33.415+0000 7f11191290c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:34.070 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:34.069+0000 7f11191290c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:34.548 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:34.548+0000 7f11191290c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:34.556 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:34.556+0000 7f11191290c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:34.556 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:34.556+0000 7f11191290c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:35.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T06:46:35.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:46:35.105 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:46:35.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd2 key to auth repository 2021-07-26T06:46:35.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:46:35.432 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2021-07-26T06:46:35.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.2 2021-07-26T06:46:35.434 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:46:35.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:35.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:46:35.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:46:35.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:46:35.456 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:35.456+0000 7ff55a3ae0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:35.458 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:35.458+0000 7ff55a3ae0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:35.459 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:35.459+0000 7ff55a3ae0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:35.460 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:35.460+0000 7ff55a3ae0c0 -1 Falling back to public interface 2021-07-26T06:46:35.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 2 2021-07-26T06:46:35.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:46:35.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:46:35.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:46:35.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:46:35.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:35.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:46:35.826 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:46:35.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:35.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:46:36.005 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:36.005+0000 7ff55a3ae0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:36.014 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:36.014+0000 7ff55a3ae0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:36.014 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:36.014+0000 7ff55a3ae0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:36.019 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:36.020+0000 7ff55a3ae0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:46:36.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:37.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:37.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:37.097 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:46:37.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:46:37.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:37.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:46:37.370 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/239676,v1:127.0.0.1:6819/239676] [v2:127.0.0.1:6820/239676,v1:127.0.0.1:6821/239676] exists,up ea4746a6-9c33-403d-9449-debe3bf0045e 2021-07-26T06:46:37.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:46:37.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:46:37.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:46:37.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:46:37.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' false = true ']' 2021-07-26T06:46:37.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 3 2021-07-26T06:46:37.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:46:37.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:46:37.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=3 2021-07-26T06:46:37.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:46:37.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:46:37.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:46:37.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:46:37.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:46:37.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:46:37.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:46:37.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:46:37.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:46:37.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:46:37.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:46:37.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:46:37.380 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:46:37.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:46:37.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:46:37.381 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:46:37.381 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:37.381 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:37.382 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:37.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:37.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:46:37.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:46:37.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:46:37.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:46:37.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:46:37.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:46:37.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:46:37.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:37.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:46:37.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:46:37.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:46:37.384 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:46:37.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=ae2ab4fe-2368-4e96-8e81-aad31b7d34ea 2021-07-26T06:46:37.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd3 ae2ab4fe-2368-4e96-8e81-aad31b7d34ea' 2021-07-26T06:46:37.385 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 ae2ab4fe-2368-4e96-8e81-aad31b7d34ea 2021-07-26T06:46:37.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:46:37.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQBNWv5gvD40FxAAqV2pFjpb0AshKmuHSHHfVQ== 2021-07-26T06:46:37.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQBNWv5gvD40FxAAqV2pFjpb0AshKmuHSHHfVQ=="}' 2021-07-26T06:46:37.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new ae2ab4fe-2368-4e96-8e81-aad31b7d34ea -i td/osd-scrub-repair/3/new.json 2021-07-26T06:46:37.766 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:46:37.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/3/new.json 2021-07-26T06:46:37.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBNWv5gvD40FxAAqV2pFjpb0AshKmuHSHHfVQ== --osd-uuid ae2ab4fe-2368-4e96-8e81-aad31b7d34ea --osd-objectstore=filestore 2021-07-26T06:46:37.803 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:37.803+0000 7fdd3ecc70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:37.816 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:37.816+0000 7fdd3ecc70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:37.816 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:37.817+0000 7fdd3ecc70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:38.312 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:38.312+0000 7fdd3ecc70c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:38.773 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:38.773+0000 7fdd3ecc70c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:38.774 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:38.774+0000 7fdd3ecc70c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:38.774 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:38.774+0000 7fdd3ecc70c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:39.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/3/keyring 2021-07-26T06:46:39.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:46:39.389 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd3 key to auth repository 2021-07-26T06:46:39.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd3 key to auth repository 2021-07-26T06:46:39.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:46:39.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.3 2021-07-26T06:46:39.731 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.3 2021-07-26T06:46:39.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:46:39.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:39.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:46:39.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:46:39.753 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:39.754+0000 7f5e6e96e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:39.788 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:39.788+0000 7f5e6e96e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:39.790 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:39.789+0000 7f5e6e96e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:39.790 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:39.790+0000 7f5e6e96e0c0 -1 Falling back to public interface 2021-07-26T06:46:40.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 3 2021-07-26T06:46:40.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:46:40.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:46:40.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:46:40.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:46:40.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:40.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:46:40.003 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:46:40.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:40.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:46:40.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:40.300 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:40.300+0000 7f5e6e96e0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:40.315 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:40.315+0000 7f5e6e96e0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:40.316 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:40.315+0000 7f5e6e96e0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:40.321 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:40.321+0000 7f5e6e96e0c0 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T06:46:41.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:41.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:41.294 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:46:41.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:46:41.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:41.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:46:41.576 INFO:tasks.workunit.client.0.smithi071.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/239960,v1:127.0.0.1:6827/239960] [v2:127.0.0.1:6828/239960,v1:127.0.0.1:6829/239960] exists,up ae2ab4fe-2368-4e96-8e81-aad31b7d34ea 2021-07-26T06:46:41.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:46:41.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:46:41.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:46:41.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:46:41.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' false = true ']' 2021-07-26T06:46:41.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 4 2021-07-26T06:46:41.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:46:41.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:46:41.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=4 2021-07-26T06:46:41.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:46:41.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/4 2021-07-26T06:46:41.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:46:41.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:46:41.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:46:41.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:46:41.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2021-07-26T06:46:41.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2021-07-26T06:46:41.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:46:41.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:46:41.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:46:41.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:46:41.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:46:41.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:46:41.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:46:41.587 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:46:41.587 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:41.587 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:41.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:41.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:41.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:46:41.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:46:41.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:46:41.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:46:41.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:46:41.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:46:41.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:46:41.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:41.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:46:41.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:46:41.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/4 2021-07-26T06:46:41.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:46:41.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=889dfefe-40ee-47a2-a1ac-3475d6f41eef 2021-07-26T06:46:41.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd3 889dfefe-40ee-47a2-a1ac-3475d6f41eef' 2021-07-26T06:46:41.592 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 889dfefe-40ee-47a2-a1ac-3475d6f41eef 2021-07-26T06:46:41.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:46:41.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQBRWv5g1X9/IxAA1QUfkTUfU54P/Epbv+XvYQ== 2021-07-26T06:46:41.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQBRWv5g1X9/IxAA1QUfkTUfU54P/Epbv+XvYQ=="}' 2021-07-26T06:46:41.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 889dfefe-40ee-47a2-a1ac-3475d6f41eef -i td/osd-scrub-repair/4/new.json 2021-07-26T06:46:42.220 INFO:tasks.workunit.client.0.smithi071.stdout:4 2021-07-26T06:46:42.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/4/new.json 2021-07-26T06:46:42.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 4 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBRWv5g1X9/IxAA1QUfkTUfU54P/Epbv+XvYQ== --osd-uuid 889dfefe-40ee-47a2-a1ac-3475d6f41eef --osd-objectstore=filestore 2021-07-26T06:46:42.258 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:42.258+0000 7f5de79f00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:42.261 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:42.262+0000 7f5de79f00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:42.262 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:42.263+0000 7f5de79f00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:43.000 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:43.000+0000 7f5de79f00c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:43.403 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:43.403+0000 7f5de79f00c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:43.412 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:43.412+0000 7f5de79f00c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:43.412 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:43.412+0000 7f5de79f00c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:44.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/4/keyring 2021-07-26T06:46:44.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:46:44.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd4 key to auth repository 2021-07-26T06:46:44.250 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd4 key to auth repository 2021-07-26T06:46:44.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:46:44.581 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.4 not found in provided keyring 2021-07-26T06:46:44.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.4 2021-07-26T06:46:44.583 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.4 2021-07-26T06:46:44.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 4 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:44.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:46:44.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:46:44.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:46:44.604 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:44.604+0000 7f64f76620c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:44.606 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:44.606+0000 7f64f76620c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:44.607 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:44.607+0000 7f64f76620c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:44.608 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:44.608+0000 7f64f76620c0 -1 Falling back to public interface 2021-07-26T06:46:44.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 4 2021-07-26T06:46:44.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:46:44.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=4 2021-07-26T06:46:44.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:46:44.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:46:44.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:44.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:46:44.973 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:46:44.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:44.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:46:45.122 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:45.122+0000 7f64f76620c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:45.129 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:45.129+0000 7f64f76620c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:45.129 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:45.129+0000 7f64f76620c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:45.136 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:45.137+0000 7f64f76620c0 -1 osd.4 0 log_to_monitors {default=true} 2021-07-26T06:46:45.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:46.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:46.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:46.246 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:46:46.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:46:46.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:46.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:46:46.521 INFO:tasks.workunit.client.0.smithi071.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/240243,v1:127.0.0.1:6835/240243] [v2:127.0.0.1:6836/240243,v1:127.0.0.1:6837/240243] exists,up 889dfefe-40ee-47a2-a1ac-3475d6f41eef 2021-07-26T06:46:46.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:46:46.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:46:46.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:46:46.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:46:46.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' false = true ']' 2021-07-26T06:46:46.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 5 2021-07-26T06:46:46.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:46:46.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:46:46.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=5 2021-07-26T06:46:46.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:46:46.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:46:46.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:46:46.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:46:46.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:46:46.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:46:46.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2021-07-26T06:46:46.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2021-07-26T06:46:46.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:46:46.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:46:46.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:46:46.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:46:46.531 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:46:46.531 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:46:46.531 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:46:46.531 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:46:46.532 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:46.532 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:46.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:46.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:46.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:46:46.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:46:46.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:46:46.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:46:46.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:46:46.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:46:46.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:46:46.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:46.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:46:46.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:46:46.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/5 2021-07-26T06:46:46.535 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:46:46.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=10f94d47-7b45-43ca-bed4-f3a0dac66239 2021-07-26T06:46:46.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd3 10f94d47-7b45-43ca-bed4-f3a0dac66239' 2021-07-26T06:46:46.536 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 10f94d47-7b45-43ca-bed4-f3a0dac66239 2021-07-26T06:46:46.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:46:46.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQBWWv5gZwQ0IBAAzLpglvzXBdD8GuZoRM1Taw== 2021-07-26T06:46:46.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQBWWv5gZwQ0IBAAzLpglvzXBdD8GuZoRM1Taw=="}' 2021-07-26T06:46:46.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 10f94d47-7b45-43ca-bed4-f3a0dac66239 -i td/osd-scrub-repair/5/new.json 2021-07-26T06:46:46.891 INFO:tasks.workunit.client.0.smithi071.stdout:5 2021-07-26T06:46:46.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/5/new.json 2021-07-26T06:46:46.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBWWv5gZwQ0IBAAzLpglvzXBdD8GuZoRM1Taw== --osd-uuid 10f94d47-7b45-43ca-bed4-f3a0dac66239 --osd-objectstore=filestore 2021-07-26T06:46:46.929 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:46.929+0000 7f216a0480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:46.930 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:46.930+0000 7f216a0480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:46.931 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:46.931+0000 7f216a0480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:47.631 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:47.632+0000 7f216a0480c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:48.327 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:48.327+0000 7f216a0480c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:48.336 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:48.336+0000 7f216a0480c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:48.336 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:48.336+0000 7f216a0480c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:48.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/5/keyring 2021-07-26T06:46:48.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:46:48.951 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd5 key to auth repository 2021-07-26T06:46:48.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd5 key to auth repository 2021-07-26T06:46:48.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:46:49.283 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.5 not found in provided keyring 2021-07-26T06:46:49.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.5 2021-07-26T06:46:49.285 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.5 2021-07-26T06:46:49.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:49.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:46:49.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:46:49.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:46:49.307 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:49.307+0000 7efcf750c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:49.308 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:49.309+0000 7efcf750c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:49.309 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:49.310+0000 7efcf750c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:49.310 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:49.311+0000 7efcf750c0c0 -1 Falling back to public interface 2021-07-26T06:46:49.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 5 2021-07-26T06:46:49.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:46:49.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:46:49.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:46:49.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:46:49.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:49.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:46:49.698 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:46:49.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:49.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:46:49.936 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:49.936+0000 7efcf750c0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:49.936 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:49.936+0000 7efcf750c0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:49.936 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:49.936+0000 7efcf750c0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:49.941 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:49.941+0000 7efcf750c0c0 -1 osd.5 0 log_to_monitors {default=true} 2021-07-26T06:46:49.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:50.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:50.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:50.972 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:46:50.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:46:50.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:50.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:46:51.255 INFO:tasks.workunit.client.0.smithi071.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/240526,v1:127.0.0.1:6843/240526] [v2:127.0.0.1:6844/240526,v1:127.0.0.1:6845/240526] exists,up 10f94d47-7b45-43ca-bed4-f3a0dac66239 2021-07-26T06:46:51.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:46:51.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:46:51.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:46:51.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:46:51.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' false = true ']' 2021-07-26T06:46:51.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 6 2021-07-26T06:46:51.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:46:51.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:46:51.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=6 2021-07-26T06:46:51.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:46:51.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/6 2021-07-26T06:46:51.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:46:51.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:46:51.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:46:51.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:46:51.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2021-07-26T06:46:51.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2021-07-26T06:46:51.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:46:51.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:46:51.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:46:51.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:46:51.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:46:51.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:46:51.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:46:51.266 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:46:51.267 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:51.267 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:51.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:51.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:51.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:46:51.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:46:51.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:46:51.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:46:51.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:46:51.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:46:51.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:46:51.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:51.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:46:51.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:46:51.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/6 2021-07-26T06:46:51.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:46:51.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=c68697ee-a313-49fc-b193-85a43f7e0da2 2021-07-26T06:46:51.272 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 c68697ee-a313-49fc-b193-85a43f7e0da2 2021-07-26T06:46:51.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd3 c68697ee-a313-49fc-b193-85a43f7e0da2' 2021-07-26T06:46:51.274 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:46:51.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQBbWv5gA5xUEBAAkDWFN1BlvBVdU3Sa2ekDSg== 2021-07-26T06:46:51.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQBbWv5gA5xUEBAAkDWFN1BlvBVdU3Sa2ekDSg=="}' 2021-07-26T06:46:51.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new c68697ee-a313-49fc-b193-85a43f7e0da2 -i td/osd-scrub-repair/6/new.json 2021-07-26T06:46:51.786 INFO:tasks.workunit.client.0.smithi071.stdout:6 2021-07-26T06:46:51.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/6/new.json 2021-07-26T06:46:51.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 6 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBbWv5gA5xUEBAAkDWFN1BlvBVdU3Sa2ekDSg== --osd-uuid c68697ee-a313-49fc-b193-85a43f7e0da2 --osd-objectstore=filestore 2021-07-26T06:46:51.822 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:51.823+0000 7f5b8c7120c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:51.927 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:51.928+0000 7f5b8c7120c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:51.928 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:51.929+0000 7f5b8c7120c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:52.818 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:52.820+0000 7f5b8c7120c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:53.499 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:53.500+0000 7f5b8c7120c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:53.500 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:53.501+0000 7f5b8c7120c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:53.500 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:53.501+0000 7f5b8c7120c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:54.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/6/keyring 2021-07-26T06:46:54.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:46:54.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd6 key to auth repository 2021-07-26T06:46:54.235 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd6 key to auth repository 2021-07-26T06:46:54.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:46:54.562 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.6 not found in provided keyring 2021-07-26T06:46:54.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.6 2021-07-26T06:46:54.564 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.6 2021-07-26T06:46:54.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 6 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:54.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:46:54.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:46:54.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:46:54.585 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:54.586+0000 7f00285c90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:54.587 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:54.588+0000 7f00285c90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:54.588 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:54.589+0000 7f00285c90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:54.589 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:54.590+0000 7f00285c90c0 -1 Falling back to public interface 2021-07-26T06:46:54.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 6 2021-07-26T06:46:54.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:46:54.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=6 2021-07-26T06:46:54.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:46:54.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:46:54.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:54.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:46:54.969 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:46:54.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:54.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.6 up' 2021-07-26T06:46:55.202 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:55.203+0000 7f00285c90c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:55.207 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:55.208+0000 7f00285c90c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:55.207 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:55.208+0000 7f00285c90c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:46:55.212 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:55.214+0000 7f00285c90c0 -1 osd.6 0 log_to_monitors {default=true} 2021-07-26T06:46:55.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:46:56.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:46:56.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:46:56.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:46:56.331 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:46:56.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:46:56.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.6 up' 2021-07-26T06:46:56.611 INFO:tasks.workunit.client.0.smithi071.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/240809,v1:127.0.0.1:6851/240809] [v2:127.0.0.1:6852/240809,v1:127.0.0.1:6853/240809] exists,up c68697ee-a313-49fc-b193-85a43f7e0da2 2021-07-26T06:46:56.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:46:56.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:46:56.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:46:56.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:46:56.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' false = true ']' 2021-07-26T06:46:56.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 7 2021-07-26T06:46:56.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:46:56.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:46:56.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=7 2021-07-26T06:46:56.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:46:56.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/7 2021-07-26T06:46:56.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:46:56.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:46:56.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:46:56.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:46:56.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2021-07-26T06:46:56.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2021-07-26T06:46:56.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:46:56.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:46:56.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:46:56.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:46:56.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:46:56.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:46:56.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:46:56.622 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:46:56.622 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:46:56.622 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:46:56.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:56.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:46:56.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:46:56.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:46:56.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:46:56.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:46:56.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:46:56.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:46:56.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:46:56.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:46:56.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:46:56.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:46:56.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/7 2021-07-26T06:46:56.626 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:46:56.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=b1fbe2d1-1610-4573-b45d-2cd9b72dea01 2021-07-26T06:46:56.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd3 b1fbe2d1-1610-4573-b45d-2cd9b72dea01' 2021-07-26T06:46:56.627 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 b1fbe2d1-1610-4573-b45d-2cd9b72dea01 2021-07-26T06:46:56.627 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:46:56.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQBgWv5ghKWlJRAAWMTlERHGLswP24eH6A83Fg== 2021-07-26T06:46:56.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQBgWv5ghKWlJRAAWMTlERHGLswP24eH6A83Fg=="}' 2021-07-26T06:46:56.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new b1fbe2d1-1610-4573-b45d-2cd9b72dea01 -i td/osd-scrub-repair/7/new.json 2021-07-26T06:46:57.001 INFO:tasks.workunit.client.0.smithi071.stdout:7 2021-07-26T06:46:57.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/7/new.json 2021-07-26T06:46:57.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 7 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBgWv5ghKWlJRAAWMTlERHGLswP24eH6A83Fg== --osd-uuid b1fbe2d1-1610-4573-b45d-2cd9b72dea01 --osd-objectstore=filestore 2021-07-26T06:46:57.038 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:57.040+0000 7f275e6ee0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:57.042 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:57.044+0000 7f275e6ee0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:57.043 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:57.044+0000 7f275e6ee0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:46:58.243 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:58.243+0000 7f275e6ee0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:59.063 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:59.064+0000 7f275e6ee0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:46:59.080 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:59.082+0000 7f275e6ee0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:59.081 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:46:59.082+0000 7f275e6ee0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:46:59.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/7/keyring 2021-07-26T06:46:59.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:46:59.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd7 key to auth repository 2021-07-26T06:46:59.736 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd7 key to auth repository 2021-07-26T06:46:59.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:47:00.079 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.7 not found in provided keyring 2021-07-26T06:47:00.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.7 2021-07-26T06:47:00.081 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.7 2021-07-26T06:47:00.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 7 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:47:00.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:47:00.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:47:00.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:47:00.102 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:00.104+0000 7fe7694c40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:00.104 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:00.105+0000 7fe7694c40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:00.105 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:00.106+0000 7fe7694c40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:00.106 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:00.107+0000 7fe7694c40c0 -1 Falling back to public interface 2021-07-26T06:47:00.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 7 2021-07-26T06:47:00.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:47:00.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=7 2021-07-26T06:47:00.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:47:00.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:47:00.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:47:00.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:47:00.594 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:47:00.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:47:00.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.7 up' 2021-07-26T06:47:00.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:47:00.989 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:00.990+0000 7fe7694c40c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:47:00.990 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:00.991+0000 7fe7694c40c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:47:00.990 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:00.991+0000 7fe7694c40c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:47:00.996 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:00.997+0000 7fe7694c40c0 -1 osd.7 0 log_to_monitors {default=true} 2021-07-26T06:47:01.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:47:01.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:47:01.986 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:47:01.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:47:01.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:47:01.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.7 up' 2021-07-26T06:47:02.266 INFO:tasks.workunit.client.0.smithi071.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/241092,v1:127.0.0.1:6859/241092] [v2:127.0.0.1:6860/241092,v1:127.0.0.1:6861/241092] exists,up b1fbe2d1-1610-4573-b45d-2cd9b72dea01 2021-07-26T06:47:02.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:47:02.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:47:02.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:47:02.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:47:02.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' false = true ']' 2021-07-26T06:47:02.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 8 2021-07-26T06:47:02.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:47:02.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:47:02.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=8 2021-07-26T06:47:02.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:47:02.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/8 2021-07-26T06:47:02.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:47:02.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:47:02.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:47:02.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:47:02.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2021-07-26T06:47:02.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2021-07-26T06:47:02.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:47:02.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:47:02.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:47:02.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:47:02.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:47:02.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:47:02.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:47:02.278 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:47:02.278 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:47:02.278 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:47:02.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:47:02.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:47:02.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:47:02.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:47:02.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:47:02.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:47:02.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:47:02.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:47:02.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:47:02.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:47:02.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:47:02.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:47:02.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/8 2021-07-26T06:47:02.282 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:47:02.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=02bbcaff-18dc-4847-ac2d-d9cb1aa047a7 2021-07-26T06:47:02.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd3 02bbcaff-18dc-4847-ac2d-d9cb1aa047a7' 2021-07-26T06:47:02.283 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 02bbcaff-18dc-4847-ac2d-d9cb1aa047a7 2021-07-26T06:47:02.283 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:47:02.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQBmWv5gXMcGERAAX9ZIm8kdVWumJGYR1q8YMg== 2021-07-26T06:47:02.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQBmWv5gXMcGERAAX9ZIm8kdVWumJGYR1q8YMg=="}' 2021-07-26T06:47:02.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 02bbcaff-18dc-4847-ac2d-d9cb1aa047a7 -i td/osd-scrub-repair/8/new.json 2021-07-26T06:47:02.842 INFO:tasks.workunit.client.0.smithi071.stdout:8 2021-07-26T06:47:02.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/8/new.json 2021-07-26T06:47:02.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 8 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBmWv5gXMcGERAAX9ZIm8kdVWumJGYR1q8YMg== --osd-uuid 02bbcaff-18dc-4847-ac2d-d9cb1aa047a7 --osd-objectstore=filestore 2021-07-26T06:47:02.880 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:02.881+0000 7fcff366c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:02.882 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:02.883+0000 7fcff366c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:02.882 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:02.884+0000 7fcff366c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:03.722 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:03.723+0000 7fcff366c0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:47:04.636 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:04.637+0000 7fcff366c0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:47:04.637 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:04.638+0000 7fcff366c0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:47:04.637 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:04.638+0000 7fcff366c0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:47:05.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/8/keyring 2021-07-26T06:47:05.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:47:05.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd8 key to auth repository 2021-07-26T06:47:05.316 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd8 key to auth repository 2021-07-26T06:47:05.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:47:05.649 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.8 not found in provided keyring 2021-07-26T06:47:05.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.8 2021-07-26T06:47:05.651 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.8 2021-07-26T06:47:05.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 8 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:47:05.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:47:05.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:47:05.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:47:05.673 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:05.674+0000 7fb92fcb50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:05.675 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:05.676+0000 7fb92fcb50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:05.676 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:05.677+0000 7fb92fcb50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:05.739 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:05.740+0000 7fb92fcb50c0 -1 Falling back to public interface 2021-07-26T06:47:06.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 8 2021-07-26T06:47:06.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:47:06.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=8 2021-07-26T06:47:06.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:47:06.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:47:06.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:47:06.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:47:06.042 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:47:06.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:47:06.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.8 up' 2021-07-26T06:47:06.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:47:06.446 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:06.447+0000 7fb92fcb50c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:47:06.447 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:06.448+0000 7fb92fcb50c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:47:06.447 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:06.448+0000 7fb92fcb50c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:47:06.452 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:06.453+0000 7fb92fcb50c0 -1 osd.8 0 log_to_monitors {default=true} 2021-07-26T06:47:07.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:47:07.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:47:07.432 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:47:07.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:47:07.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:47:07.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.8 up' 2021-07-26T06:47:07.707 INFO:tasks.workunit.client.0.smithi071.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/241375,v1:127.0.0.1:6867/241375] [v2:127.0.0.1:6868/241375,v1:127.0.0.1:6869/241375] exists,up 02bbcaff-18dc-4847-ac2d-d9cb1aa047a7 2021-07-26T06:47:07.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:47:07.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:47:07.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:47:07.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:47:07.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' false = true ']' 2021-07-26T06:47:07.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 9 2021-07-26T06:47:07.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:47:07.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:47:07.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=9 2021-07-26T06:47:07.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:47:07.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:47:07.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:47:07.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:47:07.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:47:07.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:47:07.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2021-07-26T06:47:07.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2021-07-26T06:47:07.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:47:07.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:47:07.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:47:07.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:47:07.725 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:47:07.725 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:47:07.725 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:47:07.726 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:47:07.726 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:47:07.727 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:47:07.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:47:07.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:47:07.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:47:07.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:47:07.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:47:07.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:47:07.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:47:07.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:47:07.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:47:07.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:47:07.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:47:07.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:47:07.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/9 2021-07-26T06:47:07.733 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:47:07.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=6b971543-85a5-4cad-884f-a8ced4b9b103 2021-07-26T06:47:07.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd3 6b971543-85a5-4cad-884f-a8ced4b9b103' 2021-07-26T06:47:07.735 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 6b971543-85a5-4cad-884f-a8ced4b9b103 2021-07-26T06:47:07.736 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:47:07.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQBrWv5g9dBVKxAAxFBNJ4x4NjLIG8+PaVKVgw== 2021-07-26T06:47:07.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQBrWv5g9dBVKxAAxFBNJ4x4NjLIG8+PaVKVgw=="}' 2021-07-26T06:47:07.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 6b971543-85a5-4cad-884f-a8ced4b9b103 -i td/osd-scrub-repair/9/new.json 2021-07-26T06:47:08.283 INFO:tasks.workunit.client.0.smithi071.stdout:9 2021-07-26T06:47:08.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/9/new.json 2021-07-26T06:47:08.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 9 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBrWv5g9dBVKxAAxFBNJ4x4NjLIG8+PaVKVgw== --osd-uuid 6b971543-85a5-4cad-884f-a8ced4b9b103 --osd-objectstore=filestore 2021-07-26T06:47:08.320 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:08.321+0000 7feeac0500c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:08.321 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:08.323+0000 7feeac0500c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:08.322 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:08.324+0000 7feeac0500c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:09.179 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:09.180+0000 7feeac0500c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:47:09.994 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:09.995+0000 7feeac0500c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:47:10.002 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:10.004+0000 7feeac0500c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:47:10.003 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:10.004+0000 7feeac0500c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:47:10.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/9/keyring 2021-07-26T06:47:10.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:47:10.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd9 key to auth repository 2021-07-26T06:47:10.759 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd9 key to auth repository 2021-07-26T06:47:10.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:47:11.092 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.9 not found in provided keyring 2021-07-26T06:47:11.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.9 2021-07-26T06:47:11.094 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.9 2021-07-26T06:47:11.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 9 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:47:11.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:47:11.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:47:11.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:47:11.116 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:11.117+0000 7ff2941820c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:11.118 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:11.119+0000 7ff2941820c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:11.118 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:11.120+0000 7ff2941820c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:11.120 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:11.121+0000 7ff2941820c0 -1 Falling back to public interface 2021-07-26T06:47:11.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 9 2021-07-26T06:47:11.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:47:11.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=9 2021-07-26T06:47:11.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:47:11.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:47:11.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:47:11.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:47:11.568 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:47:11.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:47:11.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:47:12.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:47:12.200 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:12.201+0000 7ff2941820c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:47:12.206 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:12.207+0000 7ff2941820c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:47:12.206 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:12.207+0000 7ff2941820c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:47:12.211 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:12.212+0000 7ff2941820c0 -1 osd.9 0 log_to_monitors {default=true} 2021-07-26T06:47:13.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:47:13.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:47:13.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:47:13.139 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:47:13.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:47:13.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:47:13.584 INFO:tasks.workunit.client.0.smithi071.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/241658,v1:127.0.0.1:6875/241658] [v2:127.0.0.1:6876/241658,v1:127.0.0.1:6877/241658] exists,up 6b971543-85a5-4cad-884f-a8ced4b9b103 2021-07-26T06:47:13.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:47:13.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:47:13.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:47:13.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: corrupt_and_repair_lrc: create_rbd_pool 2021-07-26T06:47:13.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:47:13.895 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T06:47:13.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:47:13.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:47:14.331 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T06:47:14.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:47:15.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:47:18.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: corrupt_and_repair_lrc: wait_for_clean 2021-07-26T06:47:18.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:47:18.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:47:18.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:47:18.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:47:18.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:47:18.315 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:47:18.315 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:47:18.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:47:18.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:47:18.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:47:18.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:47:18.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:47:18.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:47:18.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:47:18.422 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:47:18.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:47:18.687 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:47:18.688 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:47:18.688 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:47:18.688 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:47:18.688 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:47:18.688 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:47:18.689 INFO:tasks.workunit.client.0.smithi071.stderr:7 2021-07-26T06:47:18.689 INFO:tasks.workunit.client.0.smithi071.stderr:8 2021-07-26T06:47:18.689 INFO:tasks.workunit.client.0.smithi071.stderr:9' 2021-07-26T06:47:18.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:47:18.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:18.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:47:18.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836491 2021-07-26T06:47:18.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836491 2021-07-26T06:47:18.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491' 2021-07-26T06:47:18.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:18.849 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:47:18.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672972 2021-07-26T06:47:18.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672972 2021-07-26T06:47:18.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672972' 2021-07-26T06:47:18.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:18.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:47:19.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509450 2021-07-26T06:47:19.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509450 2021-07-26T06:47:19.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672972 2-64424509450' 2021-07-26T06:47:19.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:19.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:47:19.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345929 2021-07-26T06:47:19.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345929 2021-07-26T06:47:19.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672972 2-64424509450 3-85899345929' 2021-07-26T06:47:19.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:19.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:47:19.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182408 2021-07-26T06:47:19.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182408 2021-07-26T06:47:19.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672972 2-64424509450 3-85899345929 4-107374182408' 2021-07-26T06:47:19.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:19.319 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:47:19.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018887 2021-07-26T06:47:19.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018887 2021-07-26T06:47:19.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672972 2-64424509450 3-85899345929 4-107374182408 5-128849018887' 2021-07-26T06:47:19.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:19.560 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:47:19.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855366 2021-07-26T06:47:19.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855366 2021-07-26T06:47:19.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672972 2-64424509450 3-85899345929 4-107374182408 5-128849018887 6-150323855366' 2021-07-26T06:47:19.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:19.677 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:47:19.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691845 2021-07-26T06:47:19.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691845 2021-07-26T06:47:19.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672972 2-64424509450 3-85899345929 4-107374182408 5-128849018887 6-150323855366 7-171798691845' 2021-07-26T06:47:19.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:19.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:47:19.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528324 2021-07-26T06:47:19.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528324 2021-07-26T06:47:19.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672972 2-64424509450 3-85899345929 4-107374182408 5-128849018887 6-150323855366 7-171798691845 8-193273528324' 2021-07-26T06:47:19.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:19.927 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:47:20.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=214748364804 2021-07-26T06:47:20.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 214748364804 2021-07-26T06:47:20.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672972 2-64424509450 3-85899345929 4-107374182408 5-128849018887 6-150323855366 7-171798691845 8-193273528324 9-214748364804' 2021-07-26T06:47:20.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:20.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836491 2021-07-26T06:47:20.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:20.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:47:20.047 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836491 2021-07-26T06:47:20.048 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:20.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836491 2021-07-26T06:47:20.049 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836491 2021-07-26T06:47:20.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-07-26T06:47:20.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:47:20.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836491 -lt 21474836491 2021-07-26T06:47:20.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:20.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672972 2021-07-26T06:47:20.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:20.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:47:20.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672972 2021-07-26T06:47:20.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:20.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672972 2021-07-26T06:47:20.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2021-07-26T06:47:20.316 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672972 2021-07-26T06:47:20.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:47:20.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672972 -lt 42949672972 2021-07-26T06:47:20.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:20.663 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509450 2021-07-26T06:47:20.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:20.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:47:20.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509450 2021-07-26T06:47:20.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:20.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509450 2021-07-26T06:47:20.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2021-07-26T06:47:20.668 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509450 2021-07-26T06:47:20.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:47:20.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509450 -lt 64424509450 2021-07-26T06:47:20.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:20.954 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345929 2021-07-26T06:47:20.954 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:20.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:47:20.956 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345929 2021-07-26T06:47:20.956 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:20.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345929 2021-07-26T06:47:20.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2021-07-26T06:47:20.957 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.3 seq 85899345929 2021-07-26T06:47:20.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:47:21.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345929 -lt 85899345929 2021-07-26T06:47:21.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:21.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182408 2021-07-26T06:47:21.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:21.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:47:21.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182408 2021-07-26T06:47:21.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:21.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182408 2021-07-26T06:47:21.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182408' 2021-07-26T06:47:21.268 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.4 seq 107374182408 2021-07-26T06:47:21.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:47:21.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182408 -lt 107374182408 2021-07-26T06:47:21.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:21.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018887 2021-07-26T06:47:21.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:21.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:47:21.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018887 2021-07-26T06:47:21.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:21.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018887 2021-07-26T06:47:21.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2021-07-26T06:47:21.593 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.5 seq 128849018887 2021-07-26T06:47:21.593 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:47:21.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018887 -lt 128849018887 2021-07-26T06:47:21.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:21.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855366 2021-07-26T06:47:21.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:21.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:47:21.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855366 2021-07-26T06:47:21.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:21.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855366 2021-07-26T06:47:21.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855366' 2021-07-26T06:47:21.915 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.6 seq 150323855366 2021-07-26T06:47:21.915 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:47:22.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855366 -lt 150323855366 2021-07-26T06:47:22.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:22.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691845 2021-07-26T06:47:22.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:22.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:47:22.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691845 2021-07-26T06:47:22.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:22.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691845 2021-07-26T06:47:22.304 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.7 seq 171798691845 2021-07-26T06:47:22.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691845' 2021-07-26T06:47:22.304 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:47:22.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691845 -lt 171798691845 2021-07-26T06:47:22.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:22.614 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528324 2021-07-26T06:47:22.614 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:22.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:47:22.615 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528324 2021-07-26T06:47:22.615 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:22.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528324 2021-07-26T06:47:22.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528324' 2021-07-26T06:47:22.616 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.8 seq 193273528324 2021-07-26T06:47:22.616 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:47:22.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528324 -lt 193273528324 2021-07-26T06:47:22.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:22.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-214748364804 2021-07-26T06:47:22.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:22.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:47:22.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-214748364804 2021-07-26T06:47:22.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:22.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=214748364804 2021-07-26T06:47:22.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 214748364804' 2021-07-26T06:47:22.887 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.9 seq 214748364804 2021-07-26T06:47:22.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:47:23.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 214748364804 -lt 214748364804 2021-07-26T06:47:23.288 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:47:23.289 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:23.289 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:23.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:47:23.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:23.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:23.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:23.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:23.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:23.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:23.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:23.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:47:23.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:23.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:23.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:24.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:47:24.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:47:24.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:47:24.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2021-07-26T06:47:24.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=ecpool 2021-07-26T06:47:24.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T06:47:24.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=false 2021-07-26T06:47:24.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T06:47:24.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2021-07-26T06:47:25.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2021-07-26T06:47:25.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2021-07-26T06:47:26.267 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' created 2021-07-26T06:47:26.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:47:27.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' false = true ']' 2021-07-26T06:47:27.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T06:47:27.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:47:27.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:47:27.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:47:27.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:47:27.280 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:47:27.280 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:47:27.280 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:47:27.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:47:27.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:47:27.282 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:47:27.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:47:27.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:47:27.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:47:27.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:47:27.385 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:47:27.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:47:27.781 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:47:27.782 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:47:27.782 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:47:27.782 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:47:27.782 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:47:27.782 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:47:27.783 INFO:tasks.workunit.client.0.smithi071.stderr:7 2021-07-26T06:47:27.783 INFO:tasks.workunit.client.0.smithi071.stderr:8 2021-07-26T06:47:27.783 INFO:tasks.workunit.client.0.smithi071.stderr:9' 2021-07-26T06:47:27.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:47:27.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:27.784 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:47:27.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836494 2021-07-26T06:47:27.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836494 2021-07-26T06:47:27.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494' 2021-07-26T06:47:27.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:27.965 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:47:28.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672975 2021-07-26T06:47:28.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672975 2021-07-26T06:47:28.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672975' 2021-07-26T06:47:28.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:28.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:47:28.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509453 2021-07-26T06:47:28.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509453 2021-07-26T06:47:28.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672975 2-64424509453' 2021-07-26T06:47:28.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:28.628 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:47:28.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345932 2021-07-26T06:47:28.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345932 2021-07-26T06:47:28.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672975 2-64424509453 3-85899345932' 2021-07-26T06:47:28.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:28.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:47:29.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182411 2021-07-26T06:47:29.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182411 2021-07-26T06:47:29.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672975 2-64424509453 3-85899345932 4-107374182411' 2021-07-26T06:47:29.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:29.049 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:47:29.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018890 2021-07-26T06:47:29.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018890 2021-07-26T06:47:29.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672975 2-64424509453 3-85899345932 4-107374182411 5-128849018890' 2021-07-26T06:47:29.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:29.243 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:47:29.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855369 2021-07-26T06:47:29.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855369 2021-07-26T06:47:29.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672975 2-64424509453 3-85899345932 4-107374182411 5-128849018890 6-150323855369' 2021-07-26T06:47:29.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:29.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:47:29.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691848 2021-07-26T06:47:29.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691848 2021-07-26T06:47:29.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672975 2-64424509453 3-85899345932 4-107374182411 5-128849018890 6-150323855369 7-171798691848' 2021-07-26T06:47:29.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:29.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:47:29.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528327 2021-07-26T06:47:29.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528327 2021-07-26T06:47:29.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672975 2-64424509453 3-85899345932 4-107374182411 5-128849018890 6-150323855369 7-171798691848 8-193273528327' 2021-07-26T06:47:29.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:29.627 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:47:29.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=214748364807 2021-07-26T06:47:29.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 214748364807 2021-07-26T06:47:29.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672975 2-64424509453 3-85899345932 4-107374182411 5-128849018890 6-150323855369 7-171798691848 8-193273528327 9-214748364807' 2021-07-26T06:47:29.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:29.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836494 2021-07-26T06:47:29.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:29.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:47:29.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836494 2021-07-26T06:47:29.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:29.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836494 2021-07-26T06:47:29.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2021-07-26T06:47:29.826 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836494 2021-07-26T06:47:29.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:47:30.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836494 -lt 21474836494 2021-07-26T06:47:30.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:30.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672975 2021-07-26T06:47:30.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:30.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:47:30.090 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672975 2021-07-26T06:47:30.091 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:30.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672975 2021-07-26T06:47:30.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2021-07-26T06:47:30.092 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672975 2021-07-26T06:47:30.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:47:30.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672975 -lt 42949672975 2021-07-26T06:47:30.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:30.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509453 2021-07-26T06:47:30.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:30.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:47:30.361 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509453 2021-07-26T06:47:30.361 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:30.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509453 2021-07-26T06:47:30.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2021-07-26T06:47:30.362 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509453 2021-07-26T06:47:30.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:47:30.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509453 -lt 64424509453 2021-07-26T06:47:30.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:30.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345932 2021-07-26T06:47:30.630 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:30.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:47:30.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345932 2021-07-26T06:47:30.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:30.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345932 2021-07-26T06:47:30.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2021-07-26T06:47:30.632 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.3 seq 85899345932 2021-07-26T06:47:30.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:47:30.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345932 -lt 85899345932 2021-07-26T06:47:30.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:30.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182411 2021-07-26T06:47:30.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:30.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:47:30.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182411 2021-07-26T06:47:30.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:30.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182411 2021-07-26T06:47:30.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182411' 2021-07-26T06:47:30.942 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.4 seq 107374182411 2021-07-26T06:47:30.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:47:31.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182411 -lt 107374182411 2021-07-26T06:47:31.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:31.210 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018890 2021-07-26T06:47:31.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:31.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:47:31.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018890 2021-07-26T06:47:31.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:31.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018890 2021-07-26T06:47:31.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018890' 2021-07-26T06:47:31.213 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.5 seq 128849018890 2021-07-26T06:47:31.214 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:47:31.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018890 -lt 128849018890 2021-07-26T06:47:31.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:31.507 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855369 2021-07-26T06:47:31.507 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:31.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:47:31.508 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855369 2021-07-26T06:47:31.509 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:31.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855369 2021-07-26T06:47:31.510 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.6 seq 150323855369 2021-07-26T06:47:31.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855369' 2021-07-26T06:47:31.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:47:31.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855369 -lt 150323855369 2021-07-26T06:47:31.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:31.792 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691848 2021-07-26T06:47:31.792 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:31.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:47:31.793 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691848 2021-07-26T06:47:31.793 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:31.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691848 2021-07-26T06:47:31.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691848' 2021-07-26T06:47:31.795 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.7 seq 171798691848 2021-07-26T06:47:31.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:47:32.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691848 -lt 171798691848 2021-07-26T06:47:32.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:32.065 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528327 2021-07-26T06:47:32.065 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:32.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:47:32.066 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528327 2021-07-26T06:47:32.066 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:32.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528327 2021-07-26T06:47:32.067 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.8 seq 193273528327 2021-07-26T06:47:32.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528327' 2021-07-26T06:47:32.068 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:47:32.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528327 -lt 193273528327 2021-07-26T06:47:32.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:32.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-214748364807 2021-07-26T06:47:32.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:32.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:47:32.337 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-214748364807 2021-07-26T06:47:32.337 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:32.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=214748364807 2021-07-26T06:47:32.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 214748364807' 2021-07-26T06:47:32.339 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.9 seq 214748364807 2021-07-26T06:47:32.339 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:47:32.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 214748364807 -lt 214748364807 2021-07-26T06:47:32.644 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:47:32.645 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:32.645 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:32.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:47:32.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:32.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:32.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:32.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:32.992 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:32.992 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:32.992 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:33.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:47:33.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:33.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:33.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:33.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:47:33.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:47:33.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:47:33.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T06:47:33.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:843: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2021-07-26T06:47:33.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2021-07-26T06:47:33.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local poolname=ecpool 2021-07-26T06:47:33.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:250: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2021-07-26T06:47:33.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:47:33.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T06:47:33.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2021-07-26T06:47:33.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:47:33.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:47:33.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:47:34.225 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:47:34.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:47:34.778 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:47:34.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:47:34.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:47:34.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T06:47:35.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2021-07-26T06:47:35.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=ecpool 2021-07-26T06:47:35.072 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:47:35.072 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:47:35.072 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:47:35.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: local primary=3 2021-07-26T06:47:35.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2021-07-26T06:47:35.351 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2021-07-26T06:47:35.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=ecpool 2021-07-26T06:47:35.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: sed -e s/3// 2021-07-26T06:47:35.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=SOMETHING 2021-07-26T06:47:35.353 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:47:35.353 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T06:47:35.688 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=3 2021-07-26T06:47:35.689 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:47:35.689 INFO:tasks.workunit.client.0.smithi071.stderr:9 2021-07-26T06:47:35.689 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:47:35.690 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:47:35.690 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:47:35.690 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:47:35.691 INFO:tasks.workunit.client.0.smithi071.stderr:7' 2021-07-26T06:47:35.691 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 3 5 9 0 6 2 1 7 2021-07-26T06:47:35.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local -a osds 2021-07-26T06:47:35.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local not_primary_first=5 2021-07-26T06:47:35.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_second=9 2021-07-26T06:47:35.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2021-07-26T06:47:35.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2021-07-26T06:47:35.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2021-07-26T06:47:35.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=3 2021-07-26T06:47:35.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:47:35.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:47:35.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:47:35.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:47:35.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:47:35.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:47:35.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:47:35.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:47:35.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:47:35.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:47:35.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:47:35.699 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:47:35.699 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:47:35.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:47:35.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:47:35.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:47:35.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:47:35.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:47:35.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:47:35.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:47:35.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:47:35.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:47:35.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:47:35.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2021-07-26T06:47:37.001 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:47:37.002 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2021-07-26T06:47:37.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:47:37.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:47:37.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:47:37.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:47:37.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:47:37.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:47:37.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:47:37.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:47:37.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:47:37.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:47:37.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:47:37.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:47:37.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:47:37.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:47:37.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:47:37.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:47:37.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:47:37.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:47:37.775 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:47:37.775 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:47:37.775 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:47:37.775 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:47:37.776 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:47:37.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:47:37.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:47:37.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:47:37.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:47:37.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:47:37.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:47:37.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:47:37.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:47:37.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:47:37.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:47:37.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:47:37.778 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.3 2021-07-26T06:47:37.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:47:37.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:47:37.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:47:37.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:47:37.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:47:37.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:47:37.795 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:37.795+0000 7fa8d40730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:37.796 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:37.797+0000 7fa8d40730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:37.797 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:37.798+0000 7fa8d40730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:37.799 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:37.799+0000 7fa8d40730c0 -1 Falling back to public interface 2021-07-26T06:47:38.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:47:38.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:47:38.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:47:38.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:47:38.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:47:38.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:47:38.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:47:38.284 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:47:38.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:47:38.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:47:38.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:47:38.602 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:38.602+0000 7fa8d40730c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:47:38.602 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:38.602+0000 7fa8d40730c0 -1 journal do_read_entry(454656): bad header magic 2021-07-26T06:47:38.602 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:38.602+0000 7fa8d40730c0 -1 journal do_read_entry(454656): bad header magic 2021-07-26T06:47:38.608 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:38.609+0000 7fa8d40730c0 -1 osd.3 65 log_to_monitors {default=true} 2021-07-26T06:47:39.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:47:39.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:47:39.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:47:39.577 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:47:39.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:47:39.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:47:40.122 INFO:tasks.workunit.client.0.smithi071.stderr:osd.3 up in weight 1 up_from 69 up_thru 69 down_at 66 last_clean_interval [20,65) [v2:127.0.0.1:6826/243565,v1:127.0.0.1:6827/243565] [v2:127.0.0.1:6828/243565,v1:127.0.0.1:6829/243565] exists,up ae2ab4fe-2368-4e96-8e81-aad31b7d34ea 2021-07-26T06:47:40.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:47:40.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:47:40.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:47:40.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:47:40.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:47:40.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:47:40.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:47:40.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:47:40.124 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:47:40.124 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:47:40.124 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:47:40.125 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:47:40.125 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:47:40.125 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:47:40.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:47:40.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:47:40.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:47:40.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:47:40.218 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:47:40.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:47:40.492 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:47:40.492 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:47:40.492 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:47:40.493 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:47:40.493 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:47:40.493 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:47:40.493 INFO:tasks.workunit.client.0.smithi071.stderr:7 2021-07-26T06:47:40.494 INFO:tasks.workunit.client.0.smithi071.stderr:8 2021-07-26T06:47:40.494 INFO:tasks.workunit.client.0.smithi071.stderr:9' 2021-07-26T06:47:40.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:47:40.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:40.494 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:47:40.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836498 2021-07-26T06:47:40.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836498 2021-07-26T06:47:40.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498' 2021-07-26T06:47:40.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:40.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:47:40.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672978 2021-07-26T06:47:40.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672978 2021-07-26T06:47:40.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672978' 2021-07-26T06:47:40.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:40.735 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:47:40.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509456 2021-07-26T06:47:40.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509456 2021-07-26T06:47:40.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509456' 2021-07-26T06:47:40.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:40.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:47:40.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=296352743426 2021-07-26T06:47:40.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 296352743426 2021-07-26T06:47:40.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509456 3-296352743426' 2021-07-26T06:47:40.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:40.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:47:41.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182414 2021-07-26T06:47:41.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182414 2021-07-26T06:47:41.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509456 3-296352743426 4-107374182414' 2021-07-26T06:47:41.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:41.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:47:41.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018893 2021-07-26T06:47:41.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018893 2021-07-26T06:47:41.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509456 3-296352743426 4-107374182414 5-128849018893' 2021-07-26T06:47:41.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:41.206 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:47:41.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855372 2021-07-26T06:47:41.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855372 2021-07-26T06:47:41.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509456 3-296352743426 4-107374182414 5-128849018893 6-150323855372' 2021-07-26T06:47:41.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:41.324 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:47:41.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691851 2021-07-26T06:47:41.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691851 2021-07-26T06:47:41.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509456 3-296352743426 4-107374182414 5-128849018893 6-150323855372 7-171798691851' 2021-07-26T06:47:41.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:41.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:47:41.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528330 2021-07-26T06:47:41.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528330 2021-07-26T06:47:41.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509456 3-296352743426 4-107374182414 5-128849018893 6-150323855372 7-171798691851 8-193273528330' 2021-07-26T06:47:41.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:47:41.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:47:41.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=214748364810 2021-07-26T06:47:41.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 214748364810 2021-07-26T06:47:41.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509456 3-296352743426 4-107374182414 5-128849018893 6-150323855372 7-171798691851 8-193273528330 9-214748364810' 2021-07-26T06:47:41.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:41.678 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836498 2021-07-26T06:47:41.678 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:41.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:47:41.680 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836498 2021-07-26T06:47:41.680 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:41.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836498 2021-07-26T06:47:41.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2021-07-26T06:47:41.681 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836498 2021-07-26T06:47:41.681 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:47:41.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836498 -lt 21474836498 2021-07-26T06:47:41.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:41.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672978 2021-07-26T06:47:41.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:41.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:47:41.980 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672978 2021-07-26T06:47:41.980 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:41.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672978 2021-07-26T06:47:41.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2021-07-26T06:47:41.981 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672978 2021-07-26T06:47:41.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:47:42.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672978 -lt 42949672978 2021-07-26T06:47:42.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:42.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509456 2021-07-26T06:47:42.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:42.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:47:42.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509456 2021-07-26T06:47:42.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:42.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509456 2021-07-26T06:47:42.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2021-07-26T06:47:42.280 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509456 2021-07-26T06:47:42.280 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:47:42.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509456 -lt 64424509456 2021-07-26T06:47:42.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:42.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-296352743426 2021-07-26T06:47:42.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:42.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:47:42.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-296352743426 2021-07-26T06:47:42.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:42.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=296352743426 2021-07-26T06:47:42.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 296352743426' 2021-07-26T06:47:42.583 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 296352743426 2021-07-26T06:47:42.583 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:47:42.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 296352743426 -lt 296352743426 2021-07-26T06:47:42.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:42.851 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182414 2021-07-26T06:47:42.851 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:42.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:47:42.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182414 2021-07-26T06:47:42.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:42.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182414 2021-07-26T06:47:42.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2021-07-26T06:47:42.854 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.4 seq 107374182414 2021-07-26T06:47:42.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:47:43.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182414 -lt 107374182414 2021-07-26T06:47:43.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:43.156 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018893 2021-07-26T06:47:43.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:43.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:47:43.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018893 2021-07-26T06:47:43.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:43.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018893 2021-07-26T06:47:43.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018893' 2021-07-26T06:47:43.159 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.5 seq 128849018893 2021-07-26T06:47:43.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:47:43.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018893 -lt 128849018893 2021-07-26T06:47:43.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:43.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855372 2021-07-26T06:47:43.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:43.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:47:43.471 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855372 2021-07-26T06:47:43.471 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:43.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855372 2021-07-26T06:47:43.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855372' 2021-07-26T06:47:43.472 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.6 seq 150323855372 2021-07-26T06:47:43.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:47:43.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855372 -lt 150323855372 2021-07-26T06:47:43.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:43.927 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691851 2021-07-26T06:47:43.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:43.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:47:43.930 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691851 2021-07-26T06:47:43.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:43.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691851 2021-07-26T06:47:43.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691851' 2021-07-26T06:47:43.932 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.7 seq 171798691851 2021-07-26T06:47:43.932 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:47:44.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691852 -lt 171798691851 2021-07-26T06:47:44.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:44.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528330 2021-07-26T06:47:44.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:44.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:47:44.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528330 2021-07-26T06:47:44.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:44.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528330 2021-07-26T06:47:44.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528330' 2021-07-26T06:47:44.214 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.8 seq 193273528330 2021-07-26T06:47:44.214 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:47:44.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528331 -lt 193273528330 2021-07-26T06:47:44.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:47:44.522 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-214748364810 2021-07-26T06:47:44.522 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:47:44.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:47:44.524 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-214748364810 2021-07-26T06:47:44.524 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:47:44.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=214748364810 2021-07-26T06:47:44.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 214748364810' 2021-07-26T06:47:44.525 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.9 seq 214748364810 2021-07-26T06:47:44.526 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:47:44.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 214748364810 -lt 214748364810 2021-07-26T06:47:44.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:47:44.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:44.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:45.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:47:45.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:45.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:45.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:45.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:45.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:45.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:45.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:45.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:47:45.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:45.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:45.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:45.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:47:45.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:47:45.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:47:45.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2021-07-26T06:47:45.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:47:45.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:47:45.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:47:45.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:47:46.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2021-07-26T06:47:46.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2021-07-26T06:47:46.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:47:46.078 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:47:46.078 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:47:46.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:47:46.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:47:46.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:47:46.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:47:25.717787+0000 2021-07-26T06:47:46.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:47:46.592 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:47:46.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:47:25.717787+0000 2021-07-26T06:47:46.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:47:46.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:47:25.717787+0000 2021-07-26T06:47:46.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:47:46.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:47:46.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:47:46.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:47:46.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:47:46.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:47:46.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:47:46.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:47:46.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:25.717787+0000 '>' 2021-07-26T06:47:25.717787+0000 2021-07-26T06:47:46.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:47:47.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:47:47.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:47:47.869 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:47:47.869 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:47:47.870 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:47:47.870 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:47:47.870 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:47:48.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:25.717787+0000 '>' 2021-07-26T06:47:25.717787+0000 2021-07-26T06:47:48.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:47:49.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:47:49.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:47:49.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:47:49.142 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:47:49.142 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:47:49.142 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:47:49.142 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:47:49.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:25.717787+0000 '>' 2021-07-26T06:47:25.717787+0000 2021-07-26T06:47:49.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:47:50.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:47:50.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:47:50.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:47:50.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:47:50.416 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:47:50.416 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:47:50.416 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:47:50.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:25.717787+0000 '>' 2021-07-26T06:47:25.717787+0000 2021-07-26T06:47:50.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:47:51.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:47:51.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:47:51.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:47:51.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:47:51.697 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:47:51.697 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:47:51.697 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:47:51.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:25.717787+0000 '>' 2021-07-26T06:47:25.717787+0000 2021-07-26T06:47:51.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:47:52.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:47:52.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:47:52.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:47:52.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:47:52.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:47:52.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:47:52.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:47:53.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:25.717787+0000 '>' 2021-07-26T06:47:25.717787+0000 2021-07-26T06:47:53.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:47:54.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:47:54.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:47:54.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:47:54.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:47:54.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:47:54.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:47:54.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:47:54.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:25.717787+0000 '>' 2021-07-26T06:47:25.717787+0000 2021-07-26T06:47:54.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:47:55.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:47:55.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:47:55.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:47:55.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:47:55.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:47:55.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:47:55.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:47:55.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:50.644702+0000 '>' 2021-07-26T06:47:25.717787+0000 2021-07-26T06:47:55.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:47:55.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:47:55.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:47:55.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:47:55.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=9 2021-07-26T06:47:55.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:47:55.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:47:55.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:47:55.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:47:55.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=9 2021-07-26T06:47:55.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:47:55.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2021-07-26T06:47:55.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:47:55.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:47:55.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:47:55.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:47:55.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:47:55.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:47:55.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:47:55.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:47:55.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:47:55.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=9 2021-07-26T06:47:55.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:47:55.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:47:55.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2021-07-26T06:47:57.238 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:47:57.238 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T06:47:57.239 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:47:57.239 INFO:tasks.workunit.client.0.smithi071.stdout:hinfo_key 2021-07-26T06:47:57.239 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:47:57.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2021-07-26T06:47:57.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:47:57.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:47:57.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=9 2021-07-26T06:47:57.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:47:57.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:47:57.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:47:57.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:47:57.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:47:57.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:47:57.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2021-07-26T06:47:57.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2021-07-26T06:47:57.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:47:57.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:47:57.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:47:57.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:47:57.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:47:57.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:47:57.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:47:57.341 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:47:57.341 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:47:57.341 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:47:57.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:47:57.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:47:57.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:47:57.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:47:57.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:47:57.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:47:57.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:47:57.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:47:57.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:47:57.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:47:57.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/9 2021-07-26T06:47:57.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.9 2021-07-26T06:47:57.344 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.9 2021-07-26T06:47:57.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 9 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:47:57.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/9/whoami 2021-07-26T06:47:57.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 9 = 9 ']' 2021-07-26T06:47:57.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:47:57.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:47:57.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:47:57.359 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:57.360+0000 7f12703490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:57.361 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:57.362+0000 7f12703490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:57.362 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:57.363+0000 7f12703490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:47:57.363 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:57.364+0000 7f12703490c0 -1 Falling back to public interface 2021-07-26T06:47:57.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 9 2021-07-26T06:47:57.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:47:57.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=9 2021-07-26T06:47:57.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:47:57.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:47:57.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:47:57.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:47:57.756 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:47:57.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:47:57.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:47:58.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:47:58.256 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:58.258+0000 7f12703490c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:47:58.276 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:58.278+0000 7f12703490c0 -1 journal do_read_entry(495616): bad header magic 2021-07-26T06:47:58.277 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:58.278+0000 7f12703490c0 -1 journal do_read_entry(495616): bad header magic 2021-07-26T06:47:58.283 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:47:58.284+0000 7f12703490c0 -1 osd.9 70 log_to_monitors {default=true} 2021-07-26T06:47:59.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:47:59.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:47:59.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:47:59.258 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:47:59.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:47:59.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:47:59.629 INFO:tasks.workunit.client.0.smithi071.stderr:osd.9 up in weight 1 up_from 74 up_thru 0 down_at 71 last_clean_interval [50,70) [v2:127.0.0.1:6874/244789,v1:127.0.0.1:6875/244789] [v2:127.0.0.1:6876/244789,v1:127.0.0.1:6877/244789] exists,up 6b971543-85a5-4cad-884f-a8ced4b9b103 2021-07-26T06:47:59.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:47:59.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:47:59.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:47:59.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:47:59.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:47:59.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:47:59.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:47:59.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:47:59.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:47:59.631 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:47:59.632 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:47:59.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:47:59.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:47:59.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:47:59.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:47:59.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:47:59.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:47:59.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:47:59.724 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:48:00.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:48:00.382 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:48:00.382 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:48:00.382 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:48:00.383 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:48:00.383 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:48:00.383 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:48:00.383 INFO:tasks.workunit.client.0.smithi071.stderr:7 2021-07-26T06:48:00.383 INFO:tasks.workunit.client.0.smithi071.stderr:8 2021-07-26T06:48:00.384 INFO:tasks.workunit.client.0.smithi071.stderr:9' 2021-07-26T06:48:00.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:48:00.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:00.385 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:48:00.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836503 2021-07-26T06:48:00.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836503 2021-07-26T06:48:00.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503' 2021-07-26T06:48:00.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:00.500 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:48:00.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672983 2021-07-26T06:48:00.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672983 2021-07-26T06:48:00.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672983' 2021-07-26T06:48:00.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:00.954 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:48:01.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509461 2021-07-26T06:48:01.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509461 2021-07-26T06:48:01.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672983 2-64424509461' 2021-07-26T06:48:01.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:01.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:48:01.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=296352743431 2021-07-26T06:48:01.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 296352743431 2021-07-26T06:48:01.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672983 2-64424509461 3-296352743431' 2021-07-26T06:48:01.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:01.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:48:01.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182419 2021-07-26T06:48:01.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182419 2021-07-26T06:48:01.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672983 2-64424509461 3-296352743431 4-107374182419' 2021-07-26T06:48:01.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:01.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:48:01.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018898 2021-07-26T06:48:01.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018898 2021-07-26T06:48:01.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672983 2-64424509461 3-296352743431 4-107374182419 5-128849018898' 2021-07-26T06:48:01.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:01.423 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:48:01.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855377 2021-07-26T06:48:01.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855377 2021-07-26T06:48:01.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672983 2-64424509461 3-296352743431 4-107374182419 5-128849018898 6-150323855377' 2021-07-26T06:48:01.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:01.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:48:01.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691856 2021-07-26T06:48:01.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691856 2021-07-26T06:48:01.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672983 2-64424509461 3-296352743431 4-107374182419 5-128849018898 6-150323855377 7-171798691856' 2021-07-26T06:48:01.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:01.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:48:01.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528335 2021-07-26T06:48:01.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528335 2021-07-26T06:48:01.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672983 2-64424509461 3-296352743431 4-107374182419 5-128849018898 6-150323855377 7-171798691856 8-193273528335' 2021-07-26T06:48:01.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:01.777 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:48:01.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=317827579906 2021-07-26T06:48:01.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 317827579906 2021-07-26T06:48:01.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672983 2-64424509461 3-296352743431 4-107374182419 5-128849018898 6-150323855377 7-171798691856 8-193273528335 9-317827579906' 2021-07-26T06:48:01.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:01.906 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836503 2021-07-26T06:48:01.906 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:01.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:48:01.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836503 2021-07-26T06:48:01.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:01.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836503 2021-07-26T06:48:01.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2021-07-26T06:48:01.909 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836503 2021-07-26T06:48:01.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:48:02.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836503 -lt 21474836503 2021-07-26T06:48:02.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:02.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672983 2021-07-26T06:48:02.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:02.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:48:02.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672983 2021-07-26T06:48:02.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:02.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672983 2021-07-26T06:48:02.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2021-07-26T06:48:02.180 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672983 2021-07-26T06:48:02.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:48:02.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672983 -lt 42949672983 2021-07-26T06:48:02.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:02.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509461 2021-07-26T06:48:02.479 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:02.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:48:02.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509461 2021-07-26T06:48:02.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:02.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509461 2021-07-26T06:48:02.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2021-07-26T06:48:02.481 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509461 2021-07-26T06:48:02.482 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:48:02.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509461 -lt 64424509461 2021-07-26T06:48:02.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:02.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-296352743431 2021-07-26T06:48:02.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:02.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:48:02.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-296352743431 2021-07-26T06:48:02.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:02.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=296352743431 2021-07-26T06:48:02.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 296352743431' 2021-07-26T06:48:02.757 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 296352743431 2021-07-26T06:48:02.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:48:03.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 296352743431 -lt 296352743431 2021-07-26T06:48:03.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:03.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182419 2021-07-26T06:48:03.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:03.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:48:03.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182419 2021-07-26T06:48:03.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:03.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182419 2021-07-26T06:48:03.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182419' 2021-07-26T06:48:03.066 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.4 seq 107374182419 2021-07-26T06:48:03.066 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:48:03.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182419 -lt 107374182419 2021-07-26T06:48:03.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:03.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018898 2021-07-26T06:48:03.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:03.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:48:03.359 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018898 2021-07-26T06:48:03.359 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:03.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018898 2021-07-26T06:48:03.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018898' 2021-07-26T06:48:03.361 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.5 seq 128849018898 2021-07-26T06:48:03.361 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:48:03.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018899 -lt 128849018898 2021-07-26T06:48:03.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:03.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855377 2021-07-26T06:48:03.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:03.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:48:03.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855377 2021-07-26T06:48:03.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:03.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855377 2021-07-26T06:48:03.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855377' 2021-07-26T06:48:03.638 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.6 seq 150323855377 2021-07-26T06:48:03.638 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:48:03.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855378 -lt 150323855377 2021-07-26T06:48:03.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:03.905 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691856 2021-07-26T06:48:03.906 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:03.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:48:03.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691856 2021-07-26T06:48:03.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:03.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691856 2021-07-26T06:48:03.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691856' 2021-07-26T06:48:03.908 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.7 seq 171798691856 2021-07-26T06:48:03.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:48:04.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691857 -lt 171798691856 2021-07-26T06:48:04.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:04.230 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528335 2021-07-26T06:48:04.230 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:04.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:48:04.231 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528335 2021-07-26T06:48:04.231 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:04.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528335 2021-07-26T06:48:04.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528335' 2021-07-26T06:48:04.233 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.8 seq 193273528335 2021-07-26T06:48:04.233 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:48:04.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528336 -lt 193273528335 2021-07-26T06:48:04.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:04.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-317827579906 2021-07-26T06:48:04.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:04.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:48:04.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-317827579906 2021-07-26T06:48:04.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:04.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=317827579906 2021-07-26T06:48:04.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 317827579906' 2021-07-26T06:48:04.542 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.9 seq 317827579906 2021-07-26T06:48:04.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:48:04.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 317827579906 -lt 317827579906 2021-07-26T06:48:04.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:48:04.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:48:04.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:48:05.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:48:05.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:48:05.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:48:05.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:48:05.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:48:05.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:48:05.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:48:05.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:48:05.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:48:05.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:48:05.490 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:48:05.490 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:48:05.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 5 2021-07-26T06:48:05.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 4 '!=' -1 2021-07-26T06:48:05.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:48:05.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=4 2021-07-26T06:48:05.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:48:05.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:48:05.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:48:05.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:48:05.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:48:05.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:48:05.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:48:05.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:48:05.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:48:05.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:48:06.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:48:06.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:48:06.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:48:06.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:48:06.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:48:06.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:48:06.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:48:06.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:48:06.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:48:06.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2021-07-26T06:48:06.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2021-07-26T06:48:06.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2021-07-26T06:48:06.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=5 2021-07-26T06:48:06.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:48:06.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:48:06.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:48:06.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=5 2021-07-26T06:48:06.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:48:06.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:48:06.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:48:06.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:48:06.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=5 2021-07-26T06:48:06.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:48:06.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2021-07-26T06:48:06.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:48:06.558 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:48:06.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:48:06.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:48:06.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:48:06.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:48:06.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:48:06.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:48:06.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:48:06.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=5 2021-07-26T06:48:06.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:48:06.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:48:06.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2021-07-26T06:48:07.893 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:48:07.893 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:48:07.894 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:48:08.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2021-07-26T06:48:08.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:48:08.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:48:08.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=5 2021-07-26T06:48:08.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:48:08.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:48:08.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:48:08.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:48:08.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:48:08.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:48:08.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2021-07-26T06:48:08.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2021-07-26T06:48:08.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:48:08.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:48:08.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:48:08.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:48:08.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:48:08.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:48:08.537 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:48:08.537 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:48:08.537 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:08.537 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:48:08.537 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:48:08.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:48:08.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:48:08.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:48:08.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:48:08.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:48:08.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:48:08.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:08.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:48:08.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:48:08.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/5 2021-07-26T06:48:08.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.5 2021-07-26T06:48:08.540 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.5 2021-07-26T06:48:08.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:08.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/5/whoami 2021-07-26T06:48:08.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 5 = 5 ']' 2021-07-26T06:48:08.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:48:08.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:48:08.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:48:08.557 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:08.558+0000 7f3e7065b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:08.618 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:08.620+0000 7f3e7065b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:08.619 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:08.621+0000 7f3e7065b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:08.620 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:08.622+0000 7f3e7065b0c0 -1 Falling back to public interface 2021-07-26T06:48:08.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 5 2021-07-26T06:48:08.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:48:08.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:48:08.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:48:08.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:48:08.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:08.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:48:08.815 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:48:08.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:08.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:48:09.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:09.265 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:09.267+0000 7f3e7065b0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:48:09.283 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:09.285+0000 7f3e7065b0c0 -1 journal do_read_entry(626688): bad header magic 2021-07-26T06:48:09.284 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:09.285+0000 7f3e7065b0c0 -1 journal do_read_entry(626688): bad header magic 2021-07-26T06:48:09.290 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:09.291+0000 7f3e7065b0c0 -1 osd.5 76 log_to_monitors {default=true} 2021-07-26T06:48:10.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:10.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:10.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:48:10.265 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:48:10.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:10.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:48:10.729 INFO:tasks.workunit.client.0.smithi071.stderr:osd.5 up in weight 1 up_from 80 up_thru 66 down_at 77 last_clean_interval [30,76) [v2:127.0.0.1:6842/245806,v1:127.0.0.1:6843/245806] [v2:127.0.0.1:6844/245806,v1:127.0.0.1:6845/245806] exists,up 10f94d47-7b45-43ca-bed4-f3a0dac66239 2021-07-26T06:48:10.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:48:10.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:48:10.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:48:10.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:48:10.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:48:10.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:48:10.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:48:10.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:48:10.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:48:10.731 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:48:10.731 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:48:10.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:48:10.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:48:10.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:48:10.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:48:10.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:48:10.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:48:10.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:48:10.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:48:11.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:48:11.475 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:48:11.475 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:48:11.475 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:48:11.475 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:48:11.476 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:48:11.476 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:48:11.476 INFO:tasks.workunit.client.0.smithi071.stderr:7 2021-07-26T06:48:11.476 INFO:tasks.workunit.client.0.smithi071.stderr:8 2021-07-26T06:48:11.476 INFO:tasks.workunit.client.0.smithi071.stderr:9' 2021-07-26T06:48:11.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:48:11.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:11.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:48:11.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836506 2021-07-26T06:48:11.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836506 2021-07-26T06:48:11.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506' 2021-07-26T06:48:11.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:11.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:48:11.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672986 2021-07-26T06:48:11.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672986 2021-07-26T06:48:11.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672986' 2021-07-26T06:48:11.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:11.784 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:48:11.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509464 2021-07-26T06:48:11.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509464 2021-07-26T06:48:11.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509464' 2021-07-26T06:48:11.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:11.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:48:12.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=296352743434 2021-07-26T06:48:12.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 296352743434 2021-07-26T06:48:12.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509464 3-296352743434' 2021-07-26T06:48:12.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:12.045 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:48:12.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182423 2021-07-26T06:48:12.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182423 2021-07-26T06:48:12.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509464 3-296352743434 4-107374182423' 2021-07-26T06:48:12.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:12.162 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:48:12.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=343597383682 2021-07-26T06:48:12.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 343597383682 2021-07-26T06:48:12.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509464 3-296352743434 4-107374182423 5-343597383682' 2021-07-26T06:48:12.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:12.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:48:12.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855381 2021-07-26T06:48:12.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855381 2021-07-26T06:48:12.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509464 3-296352743434 4-107374182423 5-343597383682 6-150323855381' 2021-07-26T06:48:12.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:12.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:48:12.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691859 2021-07-26T06:48:12.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691859 2021-07-26T06:48:12.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509464 3-296352743434 4-107374182423 5-343597383682 6-150323855381 7-171798691859' 2021-07-26T06:48:12.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:12.514 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:48:12.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528338 2021-07-26T06:48:12.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528338 2021-07-26T06:48:12.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509464 3-296352743434 4-107374182423 5-343597383682 6-150323855381 7-171798691859 8-193273528338' 2021-07-26T06:48:12.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:12.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:48:12.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=317827579909 2021-07-26T06:48:12.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 317827579909 2021-07-26T06:48:12.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509464 3-296352743434 4-107374182423 5-343597383682 6-150323855381 7-171798691859 8-193273528338 9-317827579909' 2021-07-26T06:48:12.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:12.750 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836506 2021-07-26T06:48:12.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:12.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:48:12.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836506 2021-07-26T06:48:12.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:12.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836506 2021-07-26T06:48:12.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2021-07-26T06:48:12.753 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836506 2021-07-26T06:48:12.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:48:13.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836505 -lt 21474836506 2021-07-26T06:48:13.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:48:14.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:48:14.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:48:14.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836506 -lt 21474836506 2021-07-26T06:48:14.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:14.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672986 2021-07-26T06:48:14.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:14.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:48:14.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672986 2021-07-26T06:48:14.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:14.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672986 2021-07-26T06:48:14.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2021-07-26T06:48:14.302 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672986 2021-07-26T06:48:14.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:48:14.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672987 -lt 42949672986 2021-07-26T06:48:14.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:14.665 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509464 2021-07-26T06:48:14.665 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:14.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:48:14.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509464 2021-07-26T06:48:14.667 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:14.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509464 2021-07-26T06:48:14.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2021-07-26T06:48:14.668 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509464 2021-07-26T06:48:14.668 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:48:14.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509465 -lt 64424509464 2021-07-26T06:48:14.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:14.976 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-296352743434 2021-07-26T06:48:14.976 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:14.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:48:14.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-296352743434 2021-07-26T06:48:14.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:14.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=296352743434 2021-07-26T06:48:14.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 296352743434' 2021-07-26T06:48:14.979 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 296352743434 2021-07-26T06:48:14.979 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:48:15.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 296352743434 -lt 296352743434 2021-07-26T06:48:15.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:15.250 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182423 2021-07-26T06:48:15.250 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:15.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:48:15.251 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182423 2021-07-26T06:48:15.252 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:15.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182423 2021-07-26T06:48:15.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182423' 2021-07-26T06:48:15.253 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.4 seq 107374182423 2021-07-26T06:48:15.253 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:48:15.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182423 -lt 107374182423 2021-07-26T06:48:15.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:15.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-343597383682 2021-07-26T06:48:15.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:15.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:48:15.524 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-343597383682 2021-07-26T06:48:15.524 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:15.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=343597383682 2021-07-26T06:48:15.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 343597383682' 2021-07-26T06:48:15.525 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.5 seq 343597383682 2021-07-26T06:48:15.526 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:48:15.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 343597383683 -lt 343597383682 2021-07-26T06:48:15.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:15.849 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855381 2021-07-26T06:48:15.849 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:15.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:48:15.850 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855381 2021-07-26T06:48:15.850 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:15.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855381 2021-07-26T06:48:15.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855381' 2021-07-26T06:48:15.851 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.6 seq 150323855381 2021-07-26T06:48:15.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:48:16.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855381 -lt 150323855381 2021-07-26T06:48:16.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:16.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691859 2021-07-26T06:48:16.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:16.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:48:16.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691859 2021-07-26T06:48:16.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:16.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691859 2021-07-26T06:48:16.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691859' 2021-07-26T06:48:16.122 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.7 seq 171798691859 2021-07-26T06:48:16.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:48:16.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691860 -lt 171798691859 2021-07-26T06:48:16.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:16.390 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528338 2021-07-26T06:48:16.390 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:16.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:48:16.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528338 2021-07-26T06:48:16.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:16.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528338 2021-07-26T06:48:16.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528338' 2021-07-26T06:48:16.392 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.8 seq 193273528338 2021-07-26T06:48:16.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:48:16.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528339 -lt 193273528338 2021-07-26T06:48:16.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:16.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-317827579909 2021-07-26T06:48:16.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:16.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:48:16.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-317827579909 2021-07-26T06:48:16.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:16.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=317827579909 2021-07-26T06:48:16.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 317827579909' 2021-07-26T06:48:16.701 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.9 seq 317827579909 2021-07-26T06:48:16.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:48:17.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 317827579910 -lt 317827579909 2021-07-26T06:48:17.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:48:17.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:48:17.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:48:17.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:48:17.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:48:17.420 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:48:17.420 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:48:17.420 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:48:17.420 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:48:17.421 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:48:17.421 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:48:17.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:48:17.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:48:17.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:48:17.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:48:18.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:48:18.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:48:18.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:48:18.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2021-07-26T06:48:18.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:48:18.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:48:18.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:48:18.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:48:18.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2021-07-26T06:48:18.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2021-07-26T06:48:18.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:48:18.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:48:18.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:18.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:18.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:18.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:18.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:47:50.644702+0000 2021-07-26T06:48:18.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:48:18.867 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:48:18.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:47:50.644702+0000 2021-07-26T06:48:18.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:48:18.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:47:50.644702+0000 2021-07-26T06:48:18.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:48:18.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:48:18.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:48:18.878 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:48:18.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:18.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:18.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:18.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:19.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:50.644702+0000 '>' 2021-07-26T06:47:50.644702+0000 2021-07-26T06:48:19.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:48:20.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:48:20.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:48:20.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:48:20.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:20.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:20.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:20.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:20.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:50.644702+0000 '>' 2021-07-26T06:47:50.644702+0000 2021-07-26T06:48:20.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:48:21.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:48:21.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:48:21.408 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:48:21.409 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:21.409 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:21.409 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:21.409 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:21.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:50.644702+0000 '>' 2021-07-26T06:47:50.644702+0000 2021-07-26T06:48:21.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:48:22.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:48:22.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:48:22.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:48:22.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:22.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:22.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:22.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:22.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:50.644702+0000 '>' 2021-07-26T06:47:50.644702+0000 2021-07-26T06:48:22.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:48:23.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:48:23.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:48:23.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:48:23.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:23.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:23.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:23.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:24.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:50.644702+0000 '>' 2021-07-26T06:47:50.644702+0000 2021-07-26T06:48:24.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:48:25.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:48:25.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:48:25.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:48:25.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:25.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:25.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:25.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:25.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:50.644702+0000 '>' 2021-07-26T06:47:50.644702+0000 2021-07-26T06:48:25.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:48:26.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:48:26.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:48:26.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:48:26.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:26.496 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:26.496 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:26.496 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:26.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:50.644702+0000 '>' 2021-07-26T06:47:50.644702+0000 2021-07-26T06:48:26.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:48:27.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:48:27.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:48:27.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:48:27.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:27.764 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:27.764 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:27.764 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:28.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:50.644702+0000 '>' 2021-07-26T06:47:50.644702+0000 2021-07-26T06:48:28.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:48:29.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:48:29.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:48:29.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:48:29.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:29.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:29.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:29.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:29.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:47:50.644702+0000 '>' 2021-07-26T06:47:50.644702+0000 2021-07-26T06:48:29.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:48:30.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:48:30.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:48:30.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:48:30.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:30.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:30.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:30.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:30.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:48:24.503678+0000 '>' 2021-07-26T06:47:50.644702+0000 2021-07-26T06:48:30.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:48:30.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:48:30.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:48:30.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:48:30.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=9 2021-07-26T06:48:30.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:48:30.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:48:30.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:48:30.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:48:30.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=9 2021-07-26T06:48:30.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:48:30.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2021-07-26T06:48:30.571 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:48:30.571 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:48:30.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:48:30.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:48:30.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:48:30.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:48:30.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:48:30.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:48:30.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:48:30.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=9 2021-07-26T06:48:30.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:48:30.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:48:30.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2021-07-26T06:48:32.041 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:48:32.041 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T06:48:32.041 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:48:32.042 INFO:tasks.workunit.client.0.smithi071.stdout:hinfo_key 2021-07-26T06:48:32.042 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:48:32.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2021-07-26T06:48:32.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:48:32.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:48:32.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=9 2021-07-26T06:48:32.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:48:32.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:48:32.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:48:32.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:48:32.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:48:32.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:48:32.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2021-07-26T06:48:32.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2021-07-26T06:48:32.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:48:32.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:48:32.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:48:32.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:48:32.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:48:32.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:48:32.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:48:32.133 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:48:32.133 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:32.134 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:48:32.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:48:32.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:48:32.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:48:32.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:48:32.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:48:32.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:48:32.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:48:32.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:32.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:48:32.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:48:32.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/9 2021-07-26T06:48:32.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.9 2021-07-26T06:48:32.136 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.9 2021-07-26T06:48:32.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 9 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:32.137 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/9/whoami 2021-07-26T06:48:32.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 9 = 9 ']' 2021-07-26T06:48:32.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:48:32.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:48:32.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:48:32.153 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:32.153+0000 7fc666cb50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:32.155 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:32.155+0000 7fc666cb50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:32.156 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:32.156+0000 7fc666cb50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:32.157 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:32.158+0000 7fc666cb50c0 -1 Falling back to public interface 2021-07-26T06:48:32.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 9 2021-07-26T06:48:32.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:48:32.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=9 2021-07-26T06:48:32.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:48:32.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:48:32.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:32.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:48:32.751 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:48:32.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:32.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:48:33.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:33.029 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:33.029+0000 7fc666cb50c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:48:33.029 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:33.029+0000 7fc666cb50c0 -1 journal do_read_entry(692224): bad header magic 2021-07-26T06:48:33.029 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:33.029+0000 7fc666cb50c0 -1 journal do_read_entry(692224): bad header magic 2021-07-26T06:48:33.035 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:48:33.035+0000 7fc666cb50c0 -1 osd.9 82 log_to_monitors {default=true} 2021-07-26T06:48:34.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:34.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:34.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:48:34.021 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:48:34.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:34.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:48:34.672 INFO:tasks.workunit.client.0.smithi071.stderr:osd.9 up in weight 1 up_from 86 up_thru 0 down_at 83 last_clean_interval [74,82) [v2:127.0.0.1:6874/247114,v1:127.0.0.1:6875/247114] [v2:127.0.0.1:6876/247114,v1:127.0.0.1:6877/247114] exists,up 6b971543-85a5-4cad-884f-a8ced4b9b103 2021-07-26T06:48:34.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:48:34.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:48:34.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:48:34.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:48:34.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:48:34.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:48:34.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:48:34.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:48:34.675 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:48:34.675 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:48:34.675 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:48:34.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:48:34.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:48:34.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:48:34.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:48:34.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:48:34.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:48:34.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:48:34.767 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:48:35.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:48:35.039 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:48:35.040 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:48:35.040 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:48:35.040 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:48:35.041 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:48:35.041 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:48:35.042 INFO:tasks.workunit.client.0.smithi071.stderr:7 2021-07-26T06:48:35.042 INFO:tasks.workunit.client.0.smithi071.stderr:8 2021-07-26T06:48:35.042 INFO:tasks.workunit.client.0.smithi071.stderr:9' 2021-07-26T06:48:35.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:48:35.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:35.043 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:48:35.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836512 2021-07-26T06:48:35.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836512 2021-07-26T06:48:35.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836512' 2021-07-26T06:48:35.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:35.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:48:35.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672992 2021-07-26T06:48:35.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672992 2021-07-26T06:48:35.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836512 1-42949672992' 2021-07-26T06:48:35.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:35.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:48:35.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509470 2021-07-26T06:48:35.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509470 2021-07-26T06:48:35.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470' 2021-07-26T06:48:35.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:35.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:48:35.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=296352743440 2021-07-26T06:48:35.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 296352743440 2021-07-26T06:48:35.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470 3-296352743440' 2021-07-26T06:48:35.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:35.699 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:48:35.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182428 2021-07-26T06:48:35.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182428 2021-07-26T06:48:35.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470 3-296352743440 4-107374182428' 2021-07-26T06:48:35.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:35.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:48:35.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=343597383688 2021-07-26T06:48:35.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 343597383688 2021-07-26T06:48:35.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470 3-296352743440 4-107374182428 5-343597383688' 2021-07-26T06:48:35.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:35.959 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:48:36.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855386 2021-07-26T06:48:36.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855386 2021-07-26T06:48:36.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470 3-296352743440 4-107374182428 5-343597383688 6-150323855386' 2021-07-26T06:48:36.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:36.077 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:48:36.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691865 2021-07-26T06:48:36.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691865 2021-07-26T06:48:36.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470 3-296352743440 4-107374182428 5-343597383688 6-150323855386 7-171798691865' 2021-07-26T06:48:36.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:36.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:48:36.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528344 2021-07-26T06:48:36.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528344 2021-07-26T06:48:36.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470 3-296352743440 4-107374182428 5-343597383688 6-150323855386 7-171798691865 8-193273528344' 2021-07-26T06:48:36.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:36.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:48:36.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=369367187458 2021-07-26T06:48:36.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 369367187458 2021-07-26T06:48:36.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470 3-296352743440 4-107374182428 5-343597383688 6-150323855386 7-171798691865 8-193273528344 9-369367187458' 2021-07-26T06:48:36.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:36.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836512 2021-07-26T06:48:36.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:36.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:48:36.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836512 2021-07-26T06:48:36.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:36.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836512 2021-07-26T06:48:36.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2021-07-26T06:48:36.430 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836512 2021-07-26T06:48:36.430 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:48:36.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836512 -lt 21474836512 2021-07-26T06:48:36.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:36.699 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672992 2021-07-26T06:48:36.699 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:36.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:48:36.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672992 2021-07-26T06:48:36.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:36.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672992 2021-07-26T06:48:36.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2021-07-26T06:48:36.702 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672992 2021-07-26T06:48:36.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:48:37.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672992 -lt 42949672992 2021-07-26T06:48:37.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:37.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509470 2021-07-26T06:48:37.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:37.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:48:37.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509470 2021-07-26T06:48:37.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:37.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509470 2021-07-26T06:48:37.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2021-07-26T06:48:37.135 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509470 2021-07-26T06:48:37.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:48:37.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509469 -lt 64424509470 2021-07-26T06:48:37.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:48:38.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:48:38.417 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:48:38.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509471 -lt 64424509470 2021-07-26T06:48:38.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:38.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-296352743440 2021-07-26T06:48:38.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:38.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:48:38.695 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-296352743440 2021-07-26T06:48:38.695 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:38.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=296352743440 2021-07-26T06:48:38.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 296352743440' 2021-07-26T06:48:38.696 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 296352743440 2021-07-26T06:48:38.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:48:39.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 296352743440 -lt 296352743440 2021-07-26T06:48:39.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:39.004 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182428 2021-07-26T06:48:39.004 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:39.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:48:39.006 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182428 2021-07-26T06:48:39.006 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:39.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182428 2021-07-26T06:48:39.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182428' 2021-07-26T06:48:39.007 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.4 seq 107374182428 2021-07-26T06:48:39.007 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:48:39.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182429 -lt 107374182428 2021-07-26T06:48:39.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:39.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-343597383688 2021-07-26T06:48:39.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:39.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:48:39.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-343597383688 2021-07-26T06:48:39.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:39.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=343597383688 2021-07-26T06:48:39.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 343597383688' 2021-07-26T06:48:39.318 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.5 seq 343597383688 2021-07-26T06:48:39.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:48:39.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 343597383688 -lt 343597383688 2021-07-26T06:48:39.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:39.626 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855386 2021-07-26T06:48:39.626 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:39.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:48:39.627 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855386 2021-07-26T06:48:39.627 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:39.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855386 2021-07-26T06:48:39.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855386' 2021-07-26T06:48:39.629 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.6 seq 150323855386 2021-07-26T06:48:39.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:48:39.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855387 -lt 150323855386 2021-07-26T06:48:39.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:39.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691865 2021-07-26T06:48:39.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:39.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:48:39.938 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691865 2021-07-26T06:48:39.938 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:39.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691865 2021-07-26T06:48:39.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691865' 2021-07-26T06:48:39.940 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.7 seq 171798691865 2021-07-26T06:48:39.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:48:40.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691866 -lt 171798691865 2021-07-26T06:48:40.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:40.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528344 2021-07-26T06:48:40.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:40.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:48:40.250 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528344 2021-07-26T06:48:40.250 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:40.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528344 2021-07-26T06:48:40.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528344' 2021-07-26T06:48:40.251 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.8 seq 193273528344 2021-07-26T06:48:40.251 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:48:40.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528345 -lt 193273528344 2021-07-26T06:48:40.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:40.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-369367187458 2021-07-26T06:48:40.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:40.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:48:40.647 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-369367187458 2021-07-26T06:48:40.647 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:40.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=369367187458 2021-07-26T06:48:40.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 369367187458' 2021-07-26T06:48:40.649 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.9 seq 369367187458 2021-07-26T06:48:40.649 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:48:40.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 369367187459 -lt 369367187458 2021-07-26T06:48:40.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:48:40.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:48:40.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:48:41.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:48:41.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:48:41.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:48:41.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:48:41.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:48:41.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:48:41.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:48:41.280 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:48:41.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:48:41.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:48:41.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:48:41.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:48:41.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:48:41.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:48:41.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:48:41.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:48:41.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:48:41.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2021-07-26T06:48:41.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2021-07-26T06:48:41.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2021-07-26T06:48:41.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=5 2021-07-26T06:48:41.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=9 2021-07-26T06:48:41.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2021-07-26T06:48:41.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:48:41.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:48:41.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:48:41.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 248033"' 2021-07-26T06:48:41.955 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 248033' 2021-07-26T06:48:41.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2021-07-26T06:48:41.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:48:41.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:48:41.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:48:41.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 248035"' 2021-07-26T06:48:41.956 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 248035' 2021-07-26T06:48:41.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2021-07-26T06:48:41.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/248036: /' 2021-07-26T06:48:41.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 248033 248035' 2021-07-26T06:48:41.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:48:41.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:48:41.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 248033 2021-07-26T06:48:41.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2021-07-26T06:48:41.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/248038: /' 2021-07-26T06:48:43.962 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:48:43.962 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:48:43.963 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=9 2021-07-26T06:48:43.963 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:48:43.963 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2021-07-26T06:48:43.963 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:48:43.964 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:48:43.964 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=9 2021-07-26T06:48:43.965 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:48:43.965 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2021-07-26T06:48:43.966 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:48:43.966 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:48:43.966 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:48:43.967 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:48:43.967 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:48:43.967 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:48:43.967 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2021-07-26T06:48:43.968 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:48:43.968 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:48:43.968 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=9 2021-07-26T06:48:43.968 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:48:43.968 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:48:43.969 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2021-07-26T06:48:43.969 INFO:tasks.workunit.client.0.smithi071.stderr:248038: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:48:43.969 INFO:tasks.workunit.client.0.smithi071.stderr:248038: Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T06:48:43.969 INFO:tasks.workunit.client.0.smithi071.stderr:248038: remove 2#2:eb822e21:::SOMETHING:head# 2021-07-26T06:48:43.970 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2021-07-26T06:48:43.970 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:48:43.970 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:48:43.970 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=9 2021-07-26T06:48:43.970 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:48:43.971 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:48:43.971 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:48:43.971 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:48:43.971 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:48:43.972 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=5 2021-07-26T06:48:43.972 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:48:43.972 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:48:43.972 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:48:43.972 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:48:43.973 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=5 2021-07-26T06:48:43.973 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:48:43.973 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2021-07-26T06:48:43.973 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:48:43.974 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:48:43.974 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:48:43.974 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:48:43.974 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:48:43.975 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:48:43.975 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:48:43.975 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:48:43.975 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:48:43.975 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=5 2021-07-26T06:48:43.976 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:48:43.976 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:48:43.976 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2021-07-26T06:48:43.977 INFO:tasks.workunit.client.0.smithi071.stderr:248036: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:48:43.977 INFO:tasks.workunit.client.0.smithi071.stderr:248036: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:48:43.977 INFO:tasks.workunit.client.0.smithi071.stderr:248036: remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:48:43.978 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2021-07-26T06:48:43.978 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:48:43.979 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:48:43.979 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=5 2021-07-26T06:48:43.980 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:48:43.980 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:48:43.981 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:48:43.982 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:48:43.982 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:48:43.983 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:48:43.983 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2021-07-26T06:48:43.984 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2021-07-26T06:48:43.984 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:48:43.984 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:48:43.985 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:48:43.985 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:48:43.986 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:48:43.986 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:48:43.986 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:48:43.987 INFO:tasks.workunit.client.0.smithi071.stderr:248038: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:48:43.987 INFO:tasks.workunit.client.0.smithi071.stderr:248038: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:43.987 INFO:tasks.workunit.client.0.smithi071.stderr:248038: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:48:43.988 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:48:43.988 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:48:43.989 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:48:43.989 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:48:43.989 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:48:43.990 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:48:43.990 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:48:43.991 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:43.991 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:48:43.992 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:48:43.992 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/9 2021-07-26T06:48:43.992 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.9 2021-07-26T06:48:43.992 INFO:tasks.workunit.client.0.smithi071.stderr:248038: start osd.9 2021-07-26T06:48:43.993 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 9 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:48:43.993 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:48:43.993 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:48:43.993 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2021-07-26T06:48:43.994 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2021-07-26T06:48:43.994 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:48:43.994 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:48:43.994 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:48:43.994 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:48:43.995 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:48:43.995 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:48:43.995 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:48:43.995 INFO:tasks.workunit.client.0.smithi071.stderr:248036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:48:43.996 INFO:tasks.workunit.client.0.smithi071.stderr:248036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:43.996 INFO:tasks.workunit.client.0.smithi071.stderr:248036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:48:43.996 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:48:43.996 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:48:43.996 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:48:43.997 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:48:43.997 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:48:43.997 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:48:43.997 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:48:43.998 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:43.998 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:48:43.998 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:48:43.998 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/5 2021-07-26T06:48:43.999 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.5 2021-07-26T06:48:43.999 INFO:tasks.workunit.client.0.smithi071.stderr:248036: start osd.5 2021-07-26T06:48:46.280 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:46.280 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/5/whoami 2021-07-26T06:48:46.280 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 5 = 5 ']' 2021-07-26T06:48:46.281 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:48:46.281 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:48:46.281 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:48:46.281 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 2021-07-26T06:48:43.990+0000 7f8fc26310c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:46.282 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 2021-07-26T06:48:44.188+0000 7f8fc26310c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:46.282 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 2021-07-26T06:48:44.189+0000 7f8fc26310c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:46.282 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 2021-07-26T06:48:44.190+0000 7f8fc26310c0 -1 Falling back to public interface 2021-07-26T06:48:46.282 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 5 2021-07-26T06:48:46.283 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:48:46.283 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:48:46.283 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:48:46.283 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:48:46.284 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:46.284 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:48:46.284 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 0 2021-07-26T06:48:46.284 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:46.285 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:48:46.285 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:46.285 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 2021-07-26T06:48:45.169+0000 7f8fc26310c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:48:46.285 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 2021-07-26T06:48:45.170+0000 7f8fc26310c0 -1 journal do_read_entry(835584): bad header magic 2021-07-26T06:48:46.286 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 2021-07-26T06:48:45.170+0000 7f8fc26310c0 -1 journal do_read_entry(835584): bad header magic 2021-07-26T06:48:46.286 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 2021-07-26T06:48:45.176+0000 7f8fc26310c0 -1 osd.5 87 log_to_monitors {default=true} 2021-07-26T06:48:46.286 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:46.286 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:46.286 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:48:46.287 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 1 2021-07-26T06:48:46.287 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:46.287 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:48:46.287 INFO:tasks.workunit.client.0.smithi071.stderr:248036: osd.5 up in weight 1 up_from 92 up_thru 92 down_at 88 last_clean_interval [80,87) [v2:127.0.0.1:6844/248216,v1:127.0.0.1:6846/248216] [v2:127.0.0.1:6848/248216,v1:127.0.0.1:6874/248216] exists,up 10f94d47-7b45-43ca-bed4-f3a0dac66239 2021-07-26T06:48:46.288 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:48:46.288 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:48:46.288 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:48:46.288 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:48:46.289 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:48:46.289 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:46.289 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/9/whoami 2021-07-26T06:48:46.289 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 9 = 9 ']' 2021-07-26T06:48:46.290 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:48:46.290 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:48:46.290 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:48:46.290 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 2021-07-26T06:48:43.990+0000 7f421043e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:46.290 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 2021-07-26T06:48:44.187+0000 7f421043e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:46.291 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 2021-07-26T06:48:44.189+0000 7f421043e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:46.291 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 2021-07-26T06:48:44.190+0000 7f421043e0c0 -1 Falling back to public interface 2021-07-26T06:48:46.291 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 9 2021-07-26T06:48:46.291 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:48:46.292 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=9 2021-07-26T06:48:46.292 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:48:46.292 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:48:46.292 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:46.292 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:48:46.293 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 0 2021-07-26T06:48:46.293 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:46.293 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:48:46.293 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:46.294 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 2021-07-26T06:48:45.169+0000 7f421043e0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:48:46.294 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 2021-07-26T06:48:45.170+0000 7f421043e0c0 -1 journal do_read_entry(790528): bad header magic 2021-07-26T06:48:46.294 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 2021-07-26T06:48:45.170+0000 7f421043e0c0 -1 journal do_read_entry(790528): bad header magic 2021-07-26T06:48:46.294 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 2021-07-26T06:48:45.176+0000 7f421043e0c0 -1 osd.9 87 log_to_monitors {default=true} 2021-07-26T06:48:46.295 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:46.295 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:46.295 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:48:46.295 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 1 2021-07-26T06:48:46.296 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:46.296 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:48:46.296 INFO:tasks.workunit.client.0.smithi071.stderr:248038: osd.9 up in weight 1 up_from 92 up_thru 0 down_at 88 last_clean_interval [86,87) [v2:127.0.0.1:6842/248214,v1:127.0.0.1:6843/248214] [v2:127.0.0.1:6845/248214,v1:127.0.0.1:6847/248214] exists,up 6b971543-85a5-4cad-884f-a8ced4b9b103 2021-07-26T06:48:46.296 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:48:46.296 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:48:46.297 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:48:46.297 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:48:46.297 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:48:47.311 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:48:47.311 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:48:47.312 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:48:47.312 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:48:47.312 INFO:tasks.workunit.client.0.smithi071.stderr:248038: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:48:47.312 INFO:tasks.workunit.client.0.smithi071.stderr:248038: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:48:47.313 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:48:47.313 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:48:47.313 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:48:47.313 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:48:47.313 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:48:47.314 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:48:47.314 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:48:47.314 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:48:47.314 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:48:47.314 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 1 2021-07-26T06:48:47.315 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 2 2021-07-26T06:48:47.315 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 3 2021-07-26T06:48:47.315 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 4 2021-07-26T06:48:47.315 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 5 2021-07-26T06:48:47.316 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 6 2021-07-26T06:48:47.316 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 7 2021-07-26T06:48:47.316 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 8 2021-07-26T06:48:47.316 INFO:tasks.workunit.client.0.smithi071.stderr:248038: 9' 2021-07-26T06:48:47.316 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:48:47.317 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:47.317 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:48:47.317 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836516 2021-07-26T06:48:47.317 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836516 2021-07-26T06:48:47.317 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836516' 2021-07-26T06:48:47.318 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:47.318 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:48:47.318 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672995 2021-07-26T06:48:47.318 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672995 2021-07-26T06:48:47.318 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836516 1-42949672995' 2021-07-26T06:48:47.319 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:47.319 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:48:47.319 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509474 2021-07-26T06:48:47.319 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509474 2021-07-26T06:48:47.320 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836516 1-42949672995 2-64424509474' 2021-07-26T06:48:47.320 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:47.320 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:48:47.320 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=296352743443 2021-07-26T06:48:47.321 INFO:tasks.workunit.client.0.smithi071.stderr:sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:48:47.321 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:48:47.321 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:48:47.322 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:48:47.322 INFO:tasks.workunit.client.0.smithi071.stderr:248036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:48:47.322 INFO:tasks.workunit.client.0.smithi071.stderr:248036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:48:47.322 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:48:47.322 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:48:47.323 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:48:47.323 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:48:47.323 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:48:47.323 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:48:47.323 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:48:47.324 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:48:47.324 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:48:47.324 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 1 2021-07-26T06:48:47.324 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 2 2021-07-26T06:48:47.325 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 3 2021-07-26T06:48:47.325 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 4 2021-07-26T06:48:47.325 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 5 2021-07-26T06:48:47.325 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 6 2021-07-26T06:48:47.325 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 7 2021-07-26T06:48:47.326 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 8 2021-07-26T06:48:47.326 INFO:tasks.workunit.client.0.smithi071.stderr:248036: 9' 2021-07-26T06:48:47.326 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:48:47.326 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:47.327 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:48:47.327 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836515 2021-07-26T06:48:47.327 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836515 2021-07-26T06:48:47.327 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515' 2021-07-26T06:48:47.328 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:47.328 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:48:47.328 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672996 2021-07-26T06:48:47.328 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672996 2021-07-26T06:48:47.328 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672996' 2021-07-26T06:48:47.329 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:47.329 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:48:47.329 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509473 2021-07-26T06:48:47.329 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509473 2021-07-26T06:48:47.329 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672996 2-64424509473' 2021-07-26T06:48:47.330 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:47.330 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:48:48.090 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=296352743444248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 296352743443 2021-07-26T06:48:48.090 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836516 1-42949672995 2-64424509474 3-296352743443' 2021-07-26T06:48:48.091 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:48.091 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:48:48.092 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182432 2021-07-26T06:48:48.092 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182432 2021-07-26T06:48:48.093 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836516 1-42949672995 2-64424509474 3-296352743443 4-107374182432' 2021-07-26T06:48:48.093 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:48.093 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:48:48.094 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=395136991235 2021-07-26T06:48:48.094 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 395136991235 2021-07-26T06:48:48.095 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836516 1-42949672995 2-64424509474 3-296352743443 4-107374182432 5-395136991235' 2021-07-26T06:48:48.095 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:48.096 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:48:48.096 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855390 2021-07-26T06:48:48.096 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855390 2021-07-26T06:48:48.097 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836516 1-42949672995 2-64424509474 3-296352743443 4-107374182432 5-395136991235 6-150323855390' 2021-07-26T06:48:48.097 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:48.098 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:48:48.099 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691869 2021-07-26T06:48:48.099 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691869 2021-07-26T06:48:48.099 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836516 1-42949672995 2-64424509474 3-296352743443 4-107374182432 5-395136991235 6-150323855390 7-171798691869' 2021-07-26T06:48:48.099 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:48.100 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:48:48.100 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528348 2021-07-26T06:48:48.100 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528348 2021-07-26T06:48:48.100 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836516 1-42949672995 2-64424509474 3-296352743443 4-107374182432 5-395136991235 6-150323855390 7-171798691869 8-193273528348' 2021-07-26T06:48:48.101 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:48.101 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:48:48.101 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=395136991234 2021-07-26T06:48:48.101 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/ceph 2021-07-26T06:48:48.102 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 296352743444 2021-07-26T06:48:48.102 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672996 2-64424509473 3-296352743444' 2021-07-26T06:48:48.102 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:48.102 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:48:48.103 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182433 2021-07-26T06:48:48.103 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182433 2021-07-26T06:48:48.103 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672996 2-64424509473 3-296352743444 4-107374182433' 2021-07-26T06:48:48.103 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:48.103 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:48:48.104 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=395136991234 2021-07-26T06:48:48.104 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 395136991234 2021-07-26T06:48:48.104 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672996 2-64424509473 3-296352743444 4-107374182433 5-395136991234' 2021-07-26T06:48:48.104 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:48.105 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:48:48.105 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855391 2021-07-26T06:48:48.105 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855391 2021-07-26T06:48:48.105 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672996 2-64424509473 3-296352743444 4-107374182433 5-395136991234 6-150323855391' 2021-07-26T06:48:48.105 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:48.106 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:48:48.106 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691870 2021-07-26T06:48:48.106 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691870 2021-07-26T06:48:48.106 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672996 2-64424509473 3-296352743444 4-107374182433 5-395136991234 6-150323855391 7-171798691870' 2021-07-26T06:48:48.106 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:48.107 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:48:48.107 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528347 2021-07-26T06:48:48.107 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528347 2021-07-26T06:48:48.107 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672996 2-64424509473 3-296352743444 4-107374182433 5-395136991234 6-150323855391 7-171798691870 8-193273528347' 2021-07-26T06:48:48.108 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:48:48.108 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:48:48.108 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=395136991235 2021-07-26T06:48:49.007 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/ceptest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 395136991234 2021-07-26T06:48:49.007 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836516 1-42949672995 2-64424509474 3-296352743443 4-107374182432 5-395136991235 6-150323855390 7-171798691869 8-193273528348 9-395136991234' 2021-07-26T06:48:49.007 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:49.007 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836516 2021-07-26T06:48:49.008 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:49.008 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:48:49.008 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836516 2021-07-26T06:48:49.008 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:49.009 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836516 2021-07-26T06:48:49.009 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2021-07-26T06:48:49.009 INFO:tasks.workunit.client.0.smithi071.stderr:248038: waiting osd.0 seq 21474836516 2021-07-26T06:48:49.009 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:48:49.009 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836516 -lt 21474836516 2021-07-26T06:48:49.010 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:49.010 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672995 2021-07-26T06:48:49.010 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:49.010 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:48:49.011 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672995 2021-07-26T06:48:49.011 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:49.011 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672995 2021-07-26T06:48:49.011 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2021-07-26T06:48:49.012 INFO:tasks.workunit.client.0.smithi071.stderr:248038: waiting osd.1 seq 42949672995 2021-07-26T06:48:49.012 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:48:49.012 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672996 -lt 42949672995 2021-07-26T06:48:49.012 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:49.013 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509474 2021-07-26T06:48:49.013 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:49.013 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:48:49.013 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509474 2021-07-26T06:48:49.013 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:49.014 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509474 2021-07-26T06:48:49.014 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2021-07-26T06:48:49.014 INFO:tasks.workunit.client.0.smithi071.stderr:248038: waiting osd.2 seq 64424509474 2021-07-26T06:48:49.015 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:48:49.015 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509475 -lt 64424509474 2021-07-26T06:48:49.015 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flushhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 395136991235 2021-07-26T06:48:49.016 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672996 2-64424509473 3-296352743444 4-107374182433 5-395136991234 6-150323855391 7-171798691870 8-193273528347 9-395136991235' 2021-07-26T06:48:49.016 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:49.016 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836515 2021-07-26T06:48:49.016 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:49.017 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:48:49.017 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836515 2021-07-26T06:48:49.017 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:49.017 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836515 2021-07-26T06:48:49.017 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2021-07-26T06:48:49.018 INFO:tasks.workunit.client.0.smithi071.stderr:248036: waiting osd.0 seq 21474836515 2021-07-26T06:48:49.018 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:48:49.018 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836516 -lt 21474836515 2021-07-26T06:48:49.018 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:49.019 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672996 2021-07-26T06:48:49.019 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:49.019 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:48:49.019 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672996 2021-07-26T06:48:49.020 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:49.020 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672996 2021-07-26T06:48:49.020 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2021-07-26T06:48:49.021 INFO:tasks.workunit.client.0.smithi071.stderr:248036: waiting osd.1 seq 42949672996 2021-07-26T06:48:49.021 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:48:49.021 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672996 -lt 42949672996 2021-07-26T06:48:49.021 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:49.022 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509473 2021-07-26T06:48:49.022 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:49.022 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:48:49.022 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509473 2021-07-26T06:48:49.023 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:49.023 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509473 2021-07-26T06:48:49.023 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2021-07-26T06:48:49.023 INFO:tasks.workunit.client.0.smithi071.stderr:248036: waiting osd.2 seq 64424509473 2021-07-26T06:48:49.023 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:48:49.024 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509475 -lt 64424509473 2021-07-26T06:48:49.889 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:49.889 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-296352743444 2021-07-26T06:48:49.890 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:49.890 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:48:49.890 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-296352743444 2021-07-26T06:48:49.890 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:49.891 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=296352743444 2021-07-26T06:48:49.891 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 296352743444' 2021-07-26T06:48:49.891 INFO:tasks.workunit.client.0.smithi071.stderr:248036: waiting osd.3 seq 296352743444 2021-07-26T06:48:49.891 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:48:49.892 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 296352743444 -lt 296352743444 2021-07-26T06:48:49.892 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:49.892 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182433 2021-07-26T06:48:49.892 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:49.892 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:48:49.893 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182433 2021-07-26T06:48:49.893 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:49.893 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182433 2021-07-26T06:48:49.893 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182433' 2021-07-26T06:48:49.893 INFO:tasks.workunit.client.0.smithi071.stderr:248036: waiting osd.4 seq 107374182433 2021-07-26T06:48:49.894 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:48:49.894 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182433 -lt 107374182433 2021-07-26T06:48:49.894 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:49.894 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-395136991234 2021-07-26T06:48:49.894 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:49.895 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:48:49.895 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-395136991234 2021-07-26T06:48:49.895 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:49.895 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=395136991234 2021-07-26T06:48:49.895 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 395136991234' 2021-07-26T06:48:49.896 INFO:tasks.workunit.client.0.smithi071.stderr:248036: waiting osd.5 seq 395136991234 2021-07-26T06:48:49.896 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:48:49.896 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 395136991235 -lt 395136991234 2021-07-26T06:48:49.896 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:49.896 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855391 2021-07-26T06:48:49.897 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:49.897 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:48:50.084 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa_pg_stats: for s in $seqs 2021-07-26T06:48:50.085 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-296352743443 2021-07-26T06:48:50.085 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:50.085 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:48:50.085 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-296352743443 2021-07-26T06:48:50.086 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:50.086 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=296352743443 2021-07-26T06:48:50.086 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 296352743443' 2021-07-26T06:48:50.086 INFO:tasks.workunit.client.0.smithi071.stderr:248038: waiting osd.3 seq 296352743443 2021-07-26T06:48:50.087 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:48:50.087 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 296352743444 -lt 296352743443 2021-07-26T06:48:50.087 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:50.087 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182432 2021-07-26T06:48:50.088 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:50.088 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:48:50.088 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182432 2021-07-26T06:48:50.088 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:50.089 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182432 2021-07-26T06:48:50.089 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182432' 2021-07-26T06:48:50.089 INFO:tasks.workunit.client.0.smithi071.stderr:248038: waiting osd.4 seq 107374182432 2021-07-26T06:48:50.089 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:48:50.090 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182433 -lt 107374182432 2021-07-26T06:48:50.090 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:50.090 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-395136991235 2021-07-26T06:48:50.092 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:50.092 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:48:50.092 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-395136991235 2021-07-26T06:48:50.092 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:50.092 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=395136991235 2021-07-26T06:48:50.093 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 395136991235' 2021-07-26T06:48:50.093 INFO:tasks.workunit.client.0.smithi071.stderr:248038: waiting osd.5 seq 395136991235 2021-07-26T06:48:50.093 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:48:50.093 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 395136991235 -lt 395136991235 2021-07-26T06:48:50.094 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:50.094 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855390 2021-07-26T06:48:50.094 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:50.094 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:48:51.086 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa//standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855391 2021-07-26T06:48:51.087 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:51.087 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855391 2021-07-26T06:48:51.088 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855391' 2021-07-26T06:48:51.088 INFO:tasks.workunit.client.0.smithi071.stderr:248036: waiting osd.6 seq 150323855391 2021-07-26T06:48:51.088 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:48:51.089 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855391 -lt 150323855391 2021-07-26T06:48:51.089 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:51.089 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691870 2021-07-26T06:48:51.090 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:51.090 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:48:51.090 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691870 2021-07-26T06:48:51.091 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:51.091 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691870 2021-07-26T06:48:51.092 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691870' 2021-07-26T06:48:51.092 INFO:tasks.workunit.client.0.smithi071.stderr:248036: waiting osd.7 seq 171798691870 2021-07-26T06:48:51.093 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:48:51.093 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691870 -lt 171798691870 2021-07-26T06:48:51.094 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:51.094 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528347 2021-07-26T06:48:51.094 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:51.095 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:48:51.095 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528347 2021-07-26T06:48:51.096 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:51.096 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528347 2021-07-26T06:48:51.096 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528347' 2021-07-26T06:48:51.097 INFO:tasks.workunit.client.0.smithi071.stderr:248036: waiting osd.8 seq 193273528347 2021-07-26T06:48:51.097 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:48:51.097 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528349 -lt 193273528347 2021-07-26T06:48:51.098 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:51.098 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-395136991235 2021-07-26T06:48:51.099 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:51.099 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:48:51.099 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-395136991235 2021-07-26T06:48:51.100 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:51.100 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=395136991235 2021-07-26T06:48:51.101 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting ostandalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855390 2021-07-26T06:48:51.101 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:51.101 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855390 2021-07-26T06:48:51.102 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855390' 2021-07-26T06:48:51.102 INFO:tasks.workunit.client.0.smithi071.stderr:248038: waiting osd.6 seq 150323855390 2021-07-26T06:48:51.103 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:48:51.103 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855391 -lt 150323855390 2021-07-26T06:48:51.104 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:51.104 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691869 2021-07-26T06:48:51.104 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:51.105 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:48:51.105 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691869 2021-07-26T06:48:51.106 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:51.106 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691869 2021-07-26T06:48:51.107 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691869' 2021-07-26T06:48:51.107 INFO:tasks.workunit.client.0.smithi071.stderr:248038: waiting osd.7 seq 171798691869 2021-07-26T06:48:51.107 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:48:51.108 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691870 -lt 171798691869 2021-07-26T06:48:51.108 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:51.109 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528348 2021-07-26T06:48:51.109 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:51.109 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:48:51.110 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528348 2021-07-26T06:48:51.111 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:51.111 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528348 2021-07-26T06:48:51.112 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528348' 2021-07-26T06:48:51.112 INFO:tasks.workunit.client.0.smithi071.stderr:248038: waiting osd.8 seq 193273528348 2021-07-26T06:48:51.112 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:48:51.113 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528349 -lt 193273528348 2021-07-26T06:48:51.113 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:48:51.114 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-395136991234 2021-07-26T06:48:51.114 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:48:51.115 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:48:51.115 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-395136991234 2021-07-26T06:48:51.115 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:48:51.116 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=395136991234 2021-07-26T06:48:52.364 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting ossd.9 seq 395136991235' 2021-07-26T06:48:52.365 INFO:tasks.workunit.client.0.smithi071.stderr:248036: waiting osd.9 seq 395136991235 2021-07-26T06:48:52.365 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:48:52.365 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 395136991235 -lt 395136991235 2021-07-26T06:48:52.365 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:48:52.366 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:48:52.366 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:48:52.366 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:48:52.366 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:48:52.367 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:48:52.367 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:48:52.367 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:48:52.367 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:48:52.368 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:48:52.368 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:48:52.368 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:48:52.368 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:48:52.368 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:48:52.369 INFO:tasks.workunit.client.0.smithi071.stderr:248036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:48:52.369 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:48:52.369 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:48:52.369 INFO:tasks.workunit.client.0.smithi071.stderr:248036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:48:52.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:48:52.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:48:52.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 248035 2021-07-26T06:48:52.446 INFO:tasks.workunit.client.0.smithi071.stderr:d.9 seq 395136991234' 2021-07-26T06:48:52.446 INFO:tasks.workunit.client.0.smithi071.stderr:248038: waiting osd.9 seq 395136991234 2021-07-26T06:48:52.446 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:48:52.446 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 395136991235 -lt 395136991234 2021-07-26T06:48:52.447 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:48:52.447 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:48:52.447 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:48:52.447 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:48:52.448 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:48:52.448 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:48:52.448 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:48:52.448 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:48:52.448 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:48:52.449 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:48:52.449 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:48:52.449 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:48:52.449 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:48:52.450 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:48:52.450 INFO:tasks.workunit.client.0.smithi071.stderr:248038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:48:52.450 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:48:52.450 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:48:52.450 INFO:tasks.workunit.client.0.smithi071.stderr:248038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:48:52.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:48:52.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:48:52.451 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:48:52.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:48:52.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2021-07-26T06:48:52.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:48:52.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2021-07-26T06:48:52.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:48:52.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:48:52.453 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:48:52.453 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:48:52.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2021-07-26T06:48:52.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2021-07-26T06:48:52.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:48:52.717 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:48:52.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:52.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:52.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:52.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:52.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:48:24.503678+0000 2021-07-26T06:48:52.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:48:53.235 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:48:53.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:48:24.503678+0000 2021-07-26T06:48:53.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:48:53.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:48:24.503678+0000 2021-07-26T06:48:53.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:48:53.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:48:53.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:48:53.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:48:53.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:53.242 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:53.242 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:53.242 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:53.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:48:24.503678+0000 '>' 2021-07-26T06:48:24.503678+0000 2021-07-26T06:48:53.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:48:54.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:48:54.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:48:54.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:48:54.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:54.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:54.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:54.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:54.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:48:24.503678+0000 '>' 2021-07-26T06:48:24.503678+0000 2021-07-26T06:48:54.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:48:55.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:48:55.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:48:55.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:48:55.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:48:55.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:48:55.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:48:55.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:48:56.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:48:53.680531+0000 '>' 2021-07-26T06:48:24.503678+0000 2021-07-26T06:48:56.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:48:56.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2021-07-26T06:48:56.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:48:56.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:48:56.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:48:56.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 250077"' 2021-07-26T06:48:56.068 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 250077' 2021-07-26T06:48:56.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:48:56.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:48:56.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:48:56.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:48:56.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 250079"' 2021-07-26T06:48:56.070 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 250079' 2021-07-26T06:48:56.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/250080: /' 2021-07-26T06:48:56.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2021-07-26T06:48:56.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 250077 250079' 2021-07-26T06:48:56.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:48:56.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:48:56.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 250077 2021-07-26T06:48:56.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:48:56.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/250083: /' 2021-07-26T06:48:57.353 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:48:57.353 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:48:57.353 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=5 2021-07-26T06:48:57.354 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:48:57.354 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:48:57.354 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:48:57.354 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:48:57.355 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=5 2021-07-26T06:48:57.355 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:48:57.355 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2021-07-26T06:48:57.355 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:48:57.355 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:48:57.356 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:48:57.356 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:48:57.356 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:48:57.356 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:48:57.357 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:48:57.357 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:48:57.357 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:48:57.357 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=5 2021-07-26T06:48:57.357 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:48:57.358 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:48:57.358 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2021-07-26T06:48:57.358 INFO:tasks.workunit.client.0.smithi071.stderr:250080: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:48:57.358 INFO:tasks.workunit.client.0.smithi071.stderr:250080: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:48:57.359 INFO:tasks.workunit.client.0.smithi071.stderr:250080: _ 2021-07-26T06:48:57.359 INFO:tasks.workunit.client.0.smithi071.stderr:250080: hinfo_key 2021-07-26T06:48:57.359 INFO:tasks.workunit.client.0.smithi071.stderr:250080: snapset 2021-07-26T06:48:57.359 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2021-07-26T06:48:57.359 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:48:57.360 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:48:57.360 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=5 2021-07-26T06:48:57.368 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:48:57.369 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:48:57.369 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:48:57.369 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clon250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:48:57.369 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:48:57.370 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=9 2021-07-26T06:48:57.370 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:48:57.370 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:48:57.370 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:48:57.371 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:48:57.371 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=9 2021-07-26T06:48:57.371 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:48:57.371 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2021-07-26T06:48:57.371 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:48:57.372 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:48:57.372 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:48:57.372 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:48:57.372 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:48:57.373 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:48:57.373 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:48:57.373 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:48:57.373 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:48:57.373 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=9 2021-07-26T06:48:57.374 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:48:57.374 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:48:57.374 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2021-07-26T06:48:57.374 INFO:tasks.workunit.client.0.smithi071.stderr:250083: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:48:57.375 INFO:tasks.workunit.client.0.smithi071.stderr:250083: Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T06:48:57.375 INFO:tasks.workunit.client.0.smithi071.stderr:250083: _ 2021-07-26T06:48:57.375 INFO:tasks.workunit.client.0.smithi071.stderr:250083: hinfo_key 2021-07-26T06:48:57.375 INFO:tasks.workunit.client.0.smithi071.stderr:250083: snapset 2021-07-26T06:48:57.376 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2021-07-26T06:48:57.376 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:48:57.376 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:48:57.376 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=9 2021-07-26T06:48:57.376 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:48:57.377 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:48:57.377 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:48:57.378 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:48:57.378 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:48:57.378 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:48:57.378 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2021-07-26T06:48:57.379 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2021-07-26T06:48:57.379 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:48:57.379 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:48:57.379 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:48:57.380 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:48:57.380 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:48:57.380 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:48:57.380 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:48:57.380 INFO:tasks.workunit.client.0.smithi071.stderr:250080: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:48:57.381 INFO:tasks.workunit.client.0.smithi071.stderr:250080: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:57.381 INFO:tasks.workunit.client.0.smithi071.stderr:250080: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:48:57.381 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:48:57.381 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:48:57.382 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:48:57.382 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:48:57.382 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:48:57.382 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:48:57.383 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:48:57.383 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:57.383 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:48:57.383 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:48:57.383 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/5 2021-07-26T06:48:57.384 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.5 2021-07-26T06:48:57.384 INFO:tasks.workunit.client.0.smithi071.stderr:250080: start osd.5 2021-07-26T06:48:57.384 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrue.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:48:57.384 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:48:57.385 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:48:57.385 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2021-07-26T06:48:57.385 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2021-07-26T06:48:57.385 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:48:57.385 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:48:57.386 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:48:57.386 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:48:57.386 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:48:57.386 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:48:57.387 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:48:57.387 INFO:tasks.workunit.client.0.smithi071.stderr:250083: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:48:57.387 INFO:tasks.workunit.client.0.smithi071.stderr:250083: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:57.387 INFO:tasks.workunit.client.0.smithi071.stderr:250083: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:48:57.387 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:48:57.388 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:48:57.388 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:48:57.388 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:48:57.388 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:48:57.389 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:48:57.389 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:48:57.389 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:57.389 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:48:57.389 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:48:57.390 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/9 2021-07-26T06:48:57.390 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.9 2021-07-26T06:48:57.390 INFO:tasks.workunit.client.0.smithi071.stderr:250083: start osd.9 2021-07-26T06:48:59.898 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 9 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:59.898 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/9/whoami 2021-07-26T06:48:59.899 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 9 = 9 ']' 2021-07-26T06:48:59.899 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:48:59.899 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:48:59.900 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:48:59.900 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 2021-07-26T06:48:57.382+0000 7f542cfe40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:59.900 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 2021-07-26T06:48:57.452+0000 7f542cfe40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:59.900 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 2021-07-26T06:48:57.453+0000 7f542cfe40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:59.901 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 2021-07-26T06:48:57.455+0000 7f542cfe40c0 -1 Falling back to public interface 2021-07-26T06:48:59.901 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 9 2021-07-26T06:48:59.901 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:48:59.901 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=9 2021-07-26T06:48:59.902 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:48:59.902 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:48:59.903 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:59.903 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:48:59.903 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 0 2021-07-26T06:48:59.903 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:59.904 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:48:59.904 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:59.904 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 2021-07-26T06:48:58.400+0000 7f542cfe40c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:48:59.904 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 2021-07-26T06:48:58.409+0000 7f542cfe40c0 -1 journal do_read_entry(905216): bad header magic 2021-07-26T06:48:59.905 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 2021-07-26T06:48:58.409+0000 7f542cfe40c0 -1 journal do_read_entry(905216): bad header magic 2021-07-26T06:48:59.905 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 2021-07-26T06:48:58.415+0000 7f542cfe40c0 -1 osd.9 94 log_to_monitors {default=true} 2021-07-26T06:48:59.905 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:59.906 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:59.906 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:48:59.906 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 1 2021-07-26T06:48:59.906 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:59.907 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:48:59.907 INFO:tasks.workunit.client.0.smithi071.stderr:250083: osd.9 up in weight 1 up_from 98 up_thru 0 down_at 95 last_clean_interval [92,94) [v2:127.0.0.1:6842/250226,v1:127.0.0.1:6844/250226] [v2:127.0.0.1:6846/250226,v1:127.0.0.1:6848/250226] exists,up 6b971543-85a5-4cad-884f-a8ced4b9b103 2021-07-26T06:48:59.907 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:48:59.907 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:48:59.908 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:48:59.908 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:48:59.908 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:48:59.908 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cephb-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:59.909 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/5/whoami 2021-07-26T06:48:59.909 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 5 = 5 ']' 2021-07-26T06:48:59.909 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:48:59.909 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:48:59.909 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:48:59.910 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 2021-07-26T06:48:57.395+0000 7ff40b64c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:59.910 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 2021-07-26T06:48:57.452+0000 7ff40b64c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:59.910 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 2021-07-26T06:48:57.453+0000 7ff40b64c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:59.910 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 2021-07-26T06:48:57.455+0000 7ff40b64c0c0 -1 Falling back to public interface 2021-07-26T06:48:59.911 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 5 2021-07-26T06:48:59.911 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:48:59.911 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:48:59.911 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:48:59.912 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:48:59.912 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:59.912 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:48:59.912 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 0 2021-07-26T06:48:59.913 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:59.913 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:48:59.913 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:59.913 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 2021-07-26T06:48:58.400+0000 7ff40b64c0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:48:59.913 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 2021-07-26T06:48:58.408+0000 7ff40b64c0c0 -1 journal do_read_entry(958464): bad header magic 2021-07-26T06:48:59.914 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 2021-07-26T06:48:58.408+0000 7ff40b64c0c0 -1 journal do_read_entry(958464): bad header magic 2021-07-26T06:48:59.914 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 2021-07-26T06:48:58.415+0000 7ff40b64c0c0 -1 osd.5 94 log_to_monitors {default=true} 2021-07-26T06:48:59.914 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:59.915 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:59.915 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:48:59.915 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 1 2021-07-26T06:48:59.916 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:59.916 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:48:59.916 INFO:tasks.workunit.client.0.smithi071.stderr:250080: osd.5 up in weight 1 up_from 98 up_thru 98 down_at 95 last_clean_interval [92,94) [v2:127.0.0.1:6843/250229,v1:127.0.0.1:6845/250229] [v2:127.0.0.1:6847/250229,v1:127.0.0.1:6849/250229] exists,up 10f94d47-7b45-43ca-bed4-f3a0dac66239 2021-07-26T06:48:59.916 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:48:59.917 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:48:59.917 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:48:59.917 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:48:59.917 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:49:01.399 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:49:01.399 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:49:01.399 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:49:01.400 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:49:01.400 INFO:tasks.workunit.client.0.smithi071.stderr:250080: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:49:01.400 INFO:tasks.workunit.client.0.smithi071.stderr:250080: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:49:01.400 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:49:01.401 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:49:01.401 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:49:01.401 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:49:01.401 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:49:01.402 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:49:01.402 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:49:01.402 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:49:01.402 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:49:01.403 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 1 2021-07-26T06:49:01.403 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 2 2021-07-26T06:49:01.403 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 3 2021-07-26T06:49:01.403 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 4 2021-07-26T06:49:01.404 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 5 2021-07-26T06:49:01.404 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 6 2021-07-26T06:49:01.404 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 7 2021-07-26T06:49:01.404 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 8 2021-07-26T06:49:01.404 INFO:tasks.workunit.client.0.smithi071.stderr:250080: 9' 2021-07-26T06:49:01.405 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:49:01.405 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:01.405 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:49:01.405 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836520 2021-07-26T06:49:01.406 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836520 2021-07-26T06:49:01.406 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836520' 2021-07-26T06:49:01.406 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:01.406 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:49:01.407 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673000 2021-07-26T06:49:01.407 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673000 2021-07-26T06:49:01.407 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836520 1-42949673000' 2021-07-26T06:49:01.408 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:01.408 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:49:01.408 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509479 2021-07-26T06:49:01.408 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509479 2021-07-26T06:49:01.409 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836520 1-42949673000 2-64424509479' 2021-07-26T06:49:01.409 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:01.409 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:49:01.409 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=29-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:49:01.409 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:49:01.410 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:49:01.410 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:49:01.410 INFO:tasks.workunit.client.0.smithi071.stderr:250083: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:49:01.410 INFO:tasks.workunit.client.0.smithi071.stderr:250083: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:49:01.411 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:49:01.411 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:49:01.411 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:49:01.411 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:49:01.412 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:49:01.412 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:49:01.412 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:49:01.412 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:49:01.413 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:49:01.413 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 1 2021-07-26T06:49:01.413 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 2 2021-07-26T06:49:01.413 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 3 2021-07-26T06:49:01.414 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 4 2021-07-26T06:49:01.414 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 5 2021-07-26T06:49:01.414 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 6 2021-07-26T06:49:01.414 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 7 2021-07-26T06:49:01.415 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 8 2021-07-26T06:49:01.415 INFO:tasks.workunit.client.0.smithi071.stderr:250083: 9' 2021-07-26T06:49:01.415 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:49:01.416 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:01.416 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:49:01.416 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836521 2021-07-26T06:49:01.416 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836521 2021-07-26T06:49:01.416 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836521' 2021-07-26T06:49:01.417 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:01.418 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:49:01.418 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673001 2021-07-26T06:49:01.419 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673001 2021-07-26T06:49:01.419 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836521 1-42949673001' 2021-07-26T06:49:01.419 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:01.419 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:49:01.419 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509478 2021-07-26T06:49:01.420 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509478 2021-07-26T06:49:01.420 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836521 1-42949673001 2-64424509478' 2021-07-26T06:49:01.420 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:01.420 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:49:02.289 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=296352743449 2021-07-26T06:49:02.289 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 296352743449 2021-07-26T06:49:02.289 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836521 1-42949673001 2-64424509478 3-296352743449' 2021-07-26T06:49:02.290 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:02.290 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:49:02.290 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182436 2021-07-26T06:49:02.290 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182436 2021-07-26T06:49:02.291 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836521 1-42949673001 2-64424509478 3-296352743449 4-107374182436' 2021-07-26T06:49:02.291 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:02.291 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:49:02.291 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=420906795011 2021-07-26T06:49:02.292 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 420906795011 2021-07-26T06:49:02.292 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836521 1-42949673001 2-64424509478 3-296352743449 4-107374182436 5-420906795011' 2021-07-26T06:49:02.292 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:02.292 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:49:02.292 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855396 2021-07-26T06:49:02.293 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855396 2021-07-26T06:49:02.293 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836521 1-42949673001 2-64424509478 3-296352743449 4-107374182436 5-420906795011 6-150323855396' 2021-07-26T06:49:02.293 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:02.293 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:49:02.294 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691873 2021-07-26T06:49:02.294 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691873 2021-07-26T06:49:02.294 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836521 1-42949673001 2-64424509478 3-296352743449 4-107374182436 5-420906795011 6-150323855396 7-171798691873' 2021-07-26T06:49:02.294 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:02.295 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:49:02.295 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528352 2021-07-26T06:49:02.295 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528352 2021-07-26T06:49:02.295 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836521 1-42949673001 2-64424509478 3-296352743449 4-107374182436 5-420906795011 6-150323855396 7-171798691873 8-193273528352' 2021-07-26T06:49:02.295 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:02.296 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:49:02.296 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=420906795010 2021-07-26T06:49:02.296 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/u6352743448 2021-07-26T06:49:02.296 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 296352743448 2021-07-26T06:49:02.296 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836520 1-42949673000 2-64424509479 3-296352743448' 2021-07-26T06:49:02.297 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:02.297 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:49:02.297 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182437 2021-07-26T06:49:02.297 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182437 2021-07-26T06:49:02.298 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836520 1-42949673000 2-64424509479 3-296352743448 4-107374182437' 2021-07-26T06:49:02.298 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:02.298 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:49:02.298 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=420906795010 2021-07-26T06:49:02.298 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 420906795010 2021-07-26T06:49:02.299 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836520 1-42949673000 2-64424509479 3-296352743448 4-107374182437 5-420906795010' 2021-07-26T06:49:02.299 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:02.299 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:49:02.299 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855395 2021-07-26T06:49:02.300 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855395 2021-07-26T06:49:02.300 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836520 1-42949673000 2-64424509479 3-296352743448 4-107374182437 5-420906795010 6-150323855395' 2021-07-26T06:49:02.300 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:02.300 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:49:02.300 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691874 2021-07-26T06:49:02.301 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691874 2021-07-26T06:49:02.301 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836520 1-42949673000 2-64424509479 3-296352743448 4-107374182437 5-420906795010 6-150323855395 7-171798691874' 2021-07-26T06:49:02.301 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:02.301 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:49:02.302 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528353 2021-07-26T06:49:02.302 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528353 2021-07-26T06:49:02.302 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836520 1-42949673000 2-64424509479 3-296352743448 4-107374182437 5-420906795010 6-150323855395 7-171798691874 8-193273528353' 2021-07-26T06:49:02.302 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:02.302 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:49:02.303 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=420906795011 2021-07-26T06:49:03.217 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 420906795011 2021-07-26T06:49:03.218 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836520 1-42949673000 2-64424509479 3-296352743448 4-107374182437 5-420906795010 6-150323855395 7-171798691874 8-193273528353 9-420906795011' 2021-07-26T06:49:03.218 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:03.218 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836520 2021-07-26T06:49:03.218 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:03.219 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:49:03.219 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836520 2021-07-26T06:49:03.219 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:03.219 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836520 2021-07-26T06:49:03.220 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2021-07-26T06:49:03.220 INFO:tasks.workunit.client.0.smithi071.stderr:250080: waiting osd.0 seq 21474836520 2021-07-26T06:49:03.220 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:49:03.220 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836521 -lt 21474836520 2021-07-26T06:49:03.220 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:03.221 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673000 2021-07-26T06:49:03.221 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:03.221 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:49:03.221 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673000 2021-07-26T06:49:03.222 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:03.222 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673000 2021-07-26T06:49:03.222 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2021-07-26T06:49:03.222 INFO:tasks.workunit.client.0.smithi071.stderr:250080: waiting osd.1 seq 42949673000 2021-07-26T06:49:03.223 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:49:03.223 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673001 -lt 42949673000 2021-07-26T06:49:03.223 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:03.223 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509479 2021-07-26T06:49:03.224 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:03.224 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:49:03.224 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509479 2021-07-26T06:49:03.224 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:03.225 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509479 2021-07-26T06:49:03.225 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2021-07-26T06:49:03.225 INFO:tasks.workunit.client.0.smithi071.stderr:250080: waiting osd.2 seq 64424509479 2021-07-26T06:49:03.225 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:49:03.225 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509479 -lt 64424509479 2021-07-26T06:49:03.226 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 420906795010 2021-07-26T06:49:03.226 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836521 1-42949673001 2-64424509478 3-296352743449 4-107374182436 5-420906795011 6-150323855396 7-171798691873 8-193273528352 9-420906795010' 2021-07-26T06:49:03.226 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:03.227 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836521 2021-07-26T06:49:03.227 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:03.227 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:49:03.227 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836521 2021-07-26T06:49:03.228 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:03.228 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836521 2021-07-26T06:49:03.228 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2021-07-26T06:49:03.228 INFO:tasks.workunit.client.0.smithi071.stderr:250083: waiting osd.0 seq 21474836521 2021-07-26T06:49:03.228 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:49:03.229 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836521 -lt 21474836521 2021-07-26T06:49:03.229 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:03.229 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673001 2021-07-26T06:49:03.229 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:03.230 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:49:03.230 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673001 2021-07-26T06:49:03.230 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:03.230 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673001 2021-07-26T06:49:03.230 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2021-07-26T06:49:03.231 INFO:tasks.workunit.client.0.smithi071.stderr:250083: waiting osd.1 seq 42949673001 2021-07-26T06:49:03.231 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:49:03.231 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673001 -lt 42949673001 2021-07-26T06:49:03.231 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:03.232 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509478 2021-07-26T06:49:03.232 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:03.232 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:49:03.232 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509478 2021-07-26T06:49:03.232 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:03.233 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509478 2021-07-26T06:49:03.233 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2021-07-26T06:49:03.233 INFO:tasks.workunit.client.0.smithi071.stderr:250083: waiting osd.2 seq 64424509478 2021-07-26T06:49:03.233 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:49:03.234 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509479 -lt 64424509478 2021-07-26T06:49:04.136 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:04.137 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-296352743449 2021-07-26T06:49:04.137 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:04.137 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:49:04.137 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-296352743449 2021-07-26T06:49:04.138 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:04.138 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=296352743449 2021-07-26T06:49:04.138 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 296352743449' 2021-07-26T06:49:04.138 INFO:tasks.workunit.client.0.smithi071.stderr:250083: waiting osd.3 seq 296352743449 2021-07-26T06:49:04.139 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:49:04.139 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 296352743449 -lt 296352743449 2021-07-26T06:49:04.139 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:04.139 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182436 2021-07-26T06:49:04.139 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:04.140 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:49:04.140 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182436 2021-07-26T06:49:04.140 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:04.140 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182436 2021-07-26T06:49:04.141 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182436' 2021-07-26T06:49:04.141 INFO:tasks.workunit.client.0.smithi071.stderr:250083: waiting osd.4 seq 107374182436 2021-07-26T06:49:04.141 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:49:04.141 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182438 -lt 107374182436 2021-07-26T06:49:04.141 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:04.142 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-420906795011 2021-07-26T06:49:04.142 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:04.143 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:49:04.143 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-420906795011 2021-07-26T06:49:04.143 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:04.143 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=420906795011 2021-07-26T06:49:04.144 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 420906795011' 2021-07-26T06:49:04.144 INFO:tasks.workunit.client.0.smithi071.stderr:250083: waiting osd.5 seq 420906795011 2021-07-26T06:49:04.144 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:49:04.144 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 420906795011 -lt 420906795011 2021-07-26T06:49:04.144 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:04.145 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855396 2021-07-26T06:49:04.145 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:04.145 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:49:04.246 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.cl2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:04.246 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-296352743448 2021-07-26T06:49:04.247 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:04.247 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:49:04.247 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-296352743448 2021-07-26T06:49:04.247 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:04.248 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=296352743448 2021-07-26T06:49:04.248 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 296352743448' 2021-07-26T06:49:04.248 INFO:tasks.workunit.client.0.smithi071.stderr:250080: waiting osd.3 seq 296352743448 2021-07-26T06:49:04.248 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:49:04.248 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 296352743449 -lt 296352743448 2021-07-26T06:49:04.249 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:04.249 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182437 2021-07-26T06:49:04.249 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:04.249 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:49:04.250 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182437 2021-07-26T06:49:04.250 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:04.250 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182437 2021-07-26T06:49:04.250 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182437' 2021-07-26T06:49:04.251 INFO:tasks.workunit.client.0.smithi071.stderr:250080: waiting osd.4 seq 107374182437 2021-07-26T06:49:04.251 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:49:04.252 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182438 -lt 107374182437 2021-07-26T06:49:04.252 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:04.252 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-420906795010 2021-07-26T06:49:04.252 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:04.253 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:49:04.253 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-420906795010 2021-07-26T06:49:04.253 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:04.253 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=420906795010 2021-07-26T06:49:04.254 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 420906795010' 2021-07-26T06:49:04.254 INFO:tasks.workunit.client.0.smithi071.stderr:250080: waiting osd.5 seq 420906795010 2021-07-26T06:49:04.254 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:49:04.254 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 420906795011 -lt 420906795010 2021-07-26T06:49:04.255 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:04.255 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855395 2021-07-26T06:49:04.255 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:04.255 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:49:05.213 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.cient.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855396 2021-07-26T06:49:05.213 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:05.214 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855396 2021-07-26T06:49:05.214 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855396' 2021-07-26T06:49:05.214 INFO:tasks.workunit.client.0.smithi071.stderr:250083: waiting osd.6 seq 150323855396 2021-07-26T06:49:05.214 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:49:05.215 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855396 -lt 150323855396 2021-07-26T06:49:05.215 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:05.215 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691873 2021-07-26T06:49:05.215 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:05.215 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:49:05.216 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691873 2021-07-26T06:49:05.216 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:05.216 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691873 2021-07-26T06:49:05.216 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2021-07-26T06:49:05.217 INFO:tasks.workunit.client.0.smithi071.stderr:250083: waiting osd.7 seq 171798691873 2021-07-26T06:49:05.217 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:49:05.217 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691875 -lt 171798691873 2021-07-26T06:49:05.217 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:05.218 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528352 2021-07-26T06:49:05.218 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:05.218 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:49:05.218 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528352 2021-07-26T06:49:05.218 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:05.219 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528352 2021-07-26T06:49:05.219 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528352' 2021-07-26T06:49:05.219 INFO:tasks.workunit.client.0.smithi071.stderr:250083: waiting osd.8 seq 193273528352 2021-07-26T06:49:05.219 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:49:05.220 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528354 -lt 193273528352 2021-07-26T06:49:05.220 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:05.220 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-420906795010 2021-07-26T06:49:05.220 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:05.220 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:49:05.221 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-420906795010 2021-07-26T06:49:05.221 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:05.221 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=420906795010 2021-07-26T06:49:05.221 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'lient.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855395 2021-07-26T06:49:05.222 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:05.222 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855395 2021-07-26T06:49:05.222 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855395' 2021-07-26T06:49:05.222 INFO:tasks.workunit.client.0.smithi071.stderr:250080: waiting osd.6 seq 150323855395 2021-07-26T06:49:05.222 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:49:05.223 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855396 -lt 150323855395 2021-07-26T06:49:05.223 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:05.223 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691874 2021-07-26T06:49:05.223 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:05.224 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:49:05.224 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691874 2021-07-26T06:49:05.224 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:05.224 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691874 2021-07-26T06:49:05.224 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2021-07-26T06:49:05.225 INFO:tasks.workunit.client.0.smithi071.stderr:250080: waiting osd.7 seq 171798691874 2021-07-26T06:49:05.225 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:49:05.225 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691875 -lt 171798691874 2021-07-26T06:49:05.225 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:05.226 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528353 2021-07-26T06:49:05.226 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:05.226 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:49:05.226 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528353 2021-07-26T06:49:05.226 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:05.227 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528353 2021-07-26T06:49:05.227 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528353' 2021-07-26T06:49:05.227 INFO:tasks.workunit.client.0.smithi071.stderr:250080: waiting osd.8 seq 193273528353 2021-07-26T06:49:05.227 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:49:05.228 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528354 -lt 193273528353 2021-07-26T06:49:05.228 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:05.228 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-420906795011 2021-07-26T06:49:05.228 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:05.228 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:49:05.229 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-420906795011 2021-07-26T06:49:05.229 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:05.229 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=420906795011 2021-07-26T06:49:06.482 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 420906795011' 2021-07-26T06:49:06.482 INFO:tasks.workunit.client.0.smithi071.stderr:250080: waiting osd.9 seq 420906795011 2021-07-26T06:49:06.483 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:49:06.483 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 420906795011 -lt 420906795011 2021-07-26T06:49:06.483 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:49:06.483 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:06.484 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:06.484 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:49:06.484 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:49:06.484 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:49:06.484 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:49:06.485 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:49:06.485 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:49:06.485 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:49:06.485 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:49:06.486 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:49:06.486 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:49:06.486 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:06.486 INFO:tasks.workunit.client.0.smithi071.stderr:250080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:06.487 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:49:06.487 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:49:06.487 INFO:tasks.workunit.client.0.smithi071.stderr:250080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:49:06.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:49:06.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:49:06.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 250079 2021-07-26T06:49:06.489 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.9 seq 420906795010' 2021-07-26T06:49:06.489 INFO:tasks.workunit.client.0.smithi071.stderr:250083: waiting osd.9 seq 420906795010 2021-07-26T06:49:06.490 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:49:06.490 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 420906795011 -lt 420906795010 2021-07-26T06:49:06.490 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:49:06.490 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:06.491 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:06.491 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:49:06.491 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:49:06.491 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:49:06.492 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:49:06.492 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:49:06.492 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:49:06.492 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:49:06.492 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:49:06.493 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:49:06.493 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:49:06.493 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:06.493 INFO:tasks.workunit.client.0.smithi071.stderr:250083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:06.494 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:49:06.494 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:49:06.494 INFO:tasks.workunit.client.0.smithi071.stderr:250083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:49:06.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:49:06.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:49:06.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:49:06.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:49:06.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2021-07-26T06:49:06.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:49:06.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:49:06.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:49:06.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2021-07-26T06:49:06.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2021-07-26T06:49:06.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2021-07-26T06:49:06.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=3 2021-07-26T06:49:06.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=5 2021-07-26T06:49:06.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2021-07-26T06:49:06.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:49:06.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:49:06.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:49:06.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 251992"' 2021-07-26T06:49:06.519 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 251992' 2021-07-26T06:49:06.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:49:06.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:49:06.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:49:06.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 251994"' 2021-07-26T06:49:06.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:49:06.520 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 251994' 2021-07-26T06:49:06.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2021-07-26T06:49:06.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 251992 251994' 2021-07-26T06:49:06.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:49:06.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:49:06.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 251992 2021-07-26T06:49:06.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/251995: /' 2021-07-26T06:49:06.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:49:06.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/251997: /' 2021-07-26T06:49:08.846 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:49:08.847 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:49:08.847 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=5 2021-07-26T06:49:08.847 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:49:08.848 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:49:08.848 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:49:08.848 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:49:08.848 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=5 2021-07-26T06:49:08.849 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:49:08.849 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2021-07-26T06:49:08.849 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:49:08.849 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:49:08.849 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:49:08.850 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:49:08.850 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:49:08.850 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:49:08.850 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:49:08.851 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:49:08.851 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:49:08.851 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=5 2021-07-26T06:49:08.852 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:49:08.852 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:49:08.852 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2021-07-26T06:49:08.852 INFO:tasks.workunit.client.0.smithi071.stderr:251997: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:49:08.853 INFO:tasks.workunit.client.0.smithi071.stderr:251997: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:49:08.853 INFO:tasks.workunit.client.0.smithi071.stderr:251997: remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:49:08.853 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2021-07-26T06:49:08.853 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:49:08.854 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:49:08.854 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=5 2021-07-26T06:49:08.854 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:49:08.854 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:49:08.854 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:49:08.855 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:49:08.855 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:49:08.855 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:49:08.856 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:49:08.856 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:49:08.856 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:49:08.856 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:49:08.857 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:49:08.857 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:49:08.857 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:49:08.857 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:49:08.857 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:49:08.858 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:49:08.858 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:49:08.858 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:49:08.858 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:49:08.859 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:49:08.859 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:49:08.859 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:49:08.859 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:49:08.859 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:49:08.860 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:49:08.860 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2021-07-26T06:49:08.860 INFO:tasks.workunit.client.0.smithi071.stderr:251995: Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:49:08.860 INFO:tasks.workunit.client.0.smithi071.stderr:251995: remove 0#2:eb822e21:::SOMETHING:head# 2021-07-26T06:49:08.860 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:49:08.861 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:49:08.861 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:49:08.861 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:49:08.861 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:49:08.862 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:49:08.862 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:49:08.863 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ra/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:49:08.863 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:49:08.863 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:49:08.863 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2021-07-26T06:49:08.864 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2021-07-26T06:49:08.864 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:49:08.864 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:49:08.864 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:49:08.864 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:49:08.865 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:49:08.865 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:49:08.865 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:49:08.865 INFO:tasks.workunit.client.0.smithi071.stderr:251997: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:49:08.867 INFO:tasks.workunit.client.0.smithi071.stderr:251997: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:49:08.867 INFO:tasks.workunit.client.0.smithi071.stderr:251997: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:49:08.868 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:08.868 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:08.868 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:49:08.868 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:49:08.869 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:49:08.869 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:49:08.869 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:49:08.869 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:49:08.870 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:49:08.870 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:49:08.870 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/5 2021-07-26T06:49:08.870 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.5 2021-07-26T06:49:08.871 INFO:tasks.workunit.client.0.smithi071.stderr:251997: start osd.5 2021-07-26T06:49:08.871 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$tio=.99' 2021-07-26T06:49:08.871 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:49:08.872 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:49:08.872 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:49:08.872 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:49:08.872 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:49:08.873 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:49:08.873 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:49:08.873 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:49:08.874 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:49:08.875 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:49:08.875 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:49:08.876 INFO:tasks.workunit.client.0.smithi071.stderr:251995: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:49:08.876 INFO:tasks.workunit.client.0.smithi071.stderr:251995: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:49:08.876 INFO:tasks.workunit.client.0.smithi071.stderr:251995: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:49:08.877 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:08.877 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:08.878 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:49:08.878 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:49:08.879 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:49:08.879 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:49:08.879 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:49:08.880 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:49:08.880 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:49:08.881 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:49:08.881 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:49:08.881 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:49:08.882 INFO:tasks.workunit.client.0.smithi071.stderr:251995: start osd.3 2021-07-26T06:49:11.234 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:49:11.234 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:49:11.235 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:49:11.235 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:49:11.235 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:49:11.236 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:49:11.236 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 2021-07-26T06:49:08.875+0000 7f2f4434c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:11.236 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 2021-07-26T06:49:08.878+0000 7f2f4434c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:11.237 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 2021-07-26T06:49:08.879+0000 7f2f4434c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:11.237 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 2021-07-26T06:49:08.881+0000 7f2f4434c0c0 -1 Falling back to public interface 2021-07-26T06:49:11.237 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:49:11.238 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:49:11.238 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:49:11.238 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:49:11.239 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:49:11.239 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:11.240 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:49:11.240 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 0 2021-07-26T06:49:11.240 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:11.241 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:49:11.241 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:49:11.242 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 2021-07-26T06:49:09.538+0000 7f2f4434c0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:49:11.242 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 2021-07-26T06:49:09.546+0000 7f2f4434c0c0 -1 journal do_read_entry(1044480): bad header magic 2021-07-26T06:49:11.242 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 2021-07-26T06:49:09.546+0000 7f2f4434c0c0 -1 journal do_read_entry(1044480): bad header magic 2021-07-26T06:49:11.243 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 2021-07-26T06:49:09.553+0000 7f2f4434c0c0 -1 osd.3 100 log_to_monitors {default=true} 2021-07-26T06:49:11.243 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:49:11.243 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:11.243 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:49:11.244 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 1 2021-07-26T06:49:11.244 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:11.244 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:49:11.244 INFO:tasks.workunit.client.0.smithi071.stderr:251995: osd.3 up in weight 1 up_from 105 up_thru 98 down_at 101 last_clean_interval [69,100) [v2:127.0.0.1:6827/252143,v1:127.0.0.1:6829/252143] [v2:127.0.0.1:6831/252143,v1:127.0.0.1:6833/252143] exists,up ae2ab4fe-2368-4e96-8e81-aad31b7d34ea 2021-07-26T06:49:11.244 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:49:11.245 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:49:11.245 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:49:11.245 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:49:11.245 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:49:11.246 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:49:11.246 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtesname.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:49:11.247 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/5/whoami 2021-07-26T06:49:11.247 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 5 = 5 ']' 2021-07-26T06:49:11.247 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:49:11.247 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:49:11.248 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:49:11.248 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 2021-07-26T06:49:08.875+0000 7f53284800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:11.248 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 2021-07-26T06:49:08.878+0000 7f53284800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:11.248 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 2021-07-26T06:49:08.879+0000 7f53284800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:11.249 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 2021-07-26T06:49:08.880+0000 7f53284800c0 -1 Falling back to public interface 2021-07-26T06:49:11.249 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 5 2021-07-26T06:49:11.249 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:49:11.250 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:49:11.250 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:49:11.251 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:49:11.251 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:11.251 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:49:11.251 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 0 2021-07-26T06:49:11.251 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:11.252 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:49:11.252 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:49:11.252 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 2021-07-26T06:49:09.538+0000 7f53284800c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:49:11.253 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 2021-07-26T06:49:09.547+0000 7f53284800c0 -1 journal do_read_entry(1064960): bad header magic 2021-07-26T06:49:11.253 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 2021-07-26T06:49:09.547+0000 7f53284800c0 -1 journal do_read_entry(1064960): bad header magic 2021-07-26T06:49:11.253 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 2021-07-26T06:49:09.553+0000 7f53284800c0 -1 osd.5 100 log_to_monitors {default=true} 2021-07-26T06:49:11.254 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:49:11.254 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:11.254 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:49:11.254 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 1 2021-07-26T06:49:11.254 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:11.255 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:49:11.255 INFO:tasks.workunit.client.0.smithi071.stderr:251997: osd.5 up in weight 1 up_from 105 up_thru 105 down_at 101 last_clean_interval [98,100) [v2:127.0.0.1:6826/252138,v1:127.0.0.1:6828/252138] [v2:127.0.0.1:6830/252138,v1:127.0.0.1:6832/252138] exists,up 10f94d47-7b45-43ca-bed4-f3a0dac66239 2021-07-26T06:49:11.255 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:49:11.255 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:49:11.256 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:49:11.256 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:49:11.256 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:49:13.060 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:49:13.061 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:49:13.061 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:49:13.061 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:49:13.061 INFO:tasks.workunit.client.0.smithi071.stderr:251997: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:49:13.062 INFO:tasks.workunit.client.0.smithi071.stderr:251997: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:49:13.062 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:49:13.062 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:49:13.062 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:49:13.063 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:49:13.063 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:49:13.063 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:49:13.064 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:49:13.064 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:49:13.064 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:49:13.064 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 1 2021-07-26T06:49:13.064 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 2 2021-07-26T06:49:13.065 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 3 2021-07-26T06:49:13.065 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 4 2021-07-26T06:49:13.065 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 5 2021-07-26T06:49:13.065 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 6 2021-07-26T06:49:13.065 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 7 2021-07-26T06:49:13.066 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 8 2021-07-26T06:49:13.066 INFO:tasks.workunit.client.0.smithi071.stderr:251997: 9' 2021-07-26T06:49:13.066 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:49:13.067 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.067 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:49:13.067 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836524 2021-07-26T06:49:13.067 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836524 2021-07-26T06:49:13.068 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524' 2021-07-26T06:49:13.068 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.068 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:49:13.068 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673005 2021-07-26T06:49:13.069 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673005 2021-07-26T06:49:13.069 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673005' 2021-07-26T06:49:13.069 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.069 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:49:13.069 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509484 2021-07-26T06:49:13.070 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509484 2021-07-26T06:49:13.070 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673005 2-64424509484' 2021-07-26T06:49:13.070 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.070 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:49:13.071 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=45097t/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:49:13.071 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:49:13.072 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:49:13.072 INFO:tasks.workunit.client.0.smithi071.stderr:251995: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:49:13.072 INFO:tasks.workunit.client.0.smithi071.stderr:251995: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:49:13.072 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:49:13.073 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:49:13.073 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:49:13.073 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:49:13.073 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:49:13.073 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:49:13.074 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:49:13.074 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:49:13.074 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:49:13.074 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 1 2021-07-26T06:49:13.075 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 2 2021-07-26T06:49:13.075 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 3 2021-07-26T06:49:13.075 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 4 2021-07-26T06:49:13.075 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 5 2021-07-26T06:49:13.076 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 6 2021-07-26T06:49:13.076 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 7 2021-07-26T06:49:13.076 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 8 2021-07-26T06:49:13.076 INFO:tasks.workunit.client.0.smithi071.stderr:251995: 9' 2021-07-26T06:49:13.076 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:49:13.077 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.077 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:49:13.077 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836525 2021-07-26T06:49:13.078 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836525 2021-07-26T06:49:13.078 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836525' 2021-07-26T06:49:13.078 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.078 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:49:13.079 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673004 2021-07-26T06:49:13.079 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673004 2021-07-26T06:49:13.079 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836525 1-42949673004' 2021-07-26T06:49:13.079 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.080 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:49:13.080 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509483 2021-07-26T06:49:13.080 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509483 2021-07-26T06:49:13.080 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509483' 2021-07-26T06:49:13.081 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.081 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:49:13.081 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=450971566083 2021-07-26T06:49:13.852 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:20771566082 2021-07-26T06:49:13.852 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 450971566082 2021-07-26T06:49:13.852 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673005 2-64424509484 3-450971566082' 2021-07-26T06:49:13.853 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.853 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:49:13.853 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182441 2021-07-26T06:49:13.853 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182441 2021-07-26T06:49:13.854 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673005 2-64424509484 3-450971566082 4-107374182441' 2021-07-26T06:49:13.854 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.854 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:49:13.854 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=450971566083 2021-07-26T06:49:13.855 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 450971566083 2021-07-26T06:49:13.855 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673005 2-64424509484 3-450971566082 4-107374182441 5-450971566083' 2021-07-26T06:49:13.855 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.855 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:49:13.856 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855399 2021-07-26T06:49:13.856 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855399 2021-07-26T06:49:13.856 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673005 2-64424509484 3-450971566082 4-107374182441 5-450971566083 6-150323855399' 2021-07-26T06:49:13.857 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.857 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:49:13.857 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691879 2021-07-26T06:49:13.857 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691879 2021-07-26T06:49:13.857 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673005 2-64424509484 3-450971566082 4-107374182441 5-450971566083 6-150323855399 7-171798691879' 2021-07-26T06:49:13.858 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.858 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:49:13.858 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528357 2021-07-26T06:49:13.858 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528357 2021-07-26T06:49:13.859 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673005 2-64424509484 3-450971566082 4-107374182441 5-450971566083 6-150323855399 7-171798691879 8-193273528357' 2021-07-26T06:49:13.859 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.859 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:49:13.859 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=420906795015 2021-07-26T06:49:13.860 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubu: flush_pg_stats: test -z 450971566083 2021-07-26T06:49:13.860 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509483 3-450971566083' 2021-07-26T06:49:13.860 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.860 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:49:13.860 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182442 2021-07-26T06:49:13.861 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182442 2021-07-26T06:49:13.861 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509483 3-450971566083 4-107374182442' 2021-07-26T06:49:13.861 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.861 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:49:13.861 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=450971566082 2021-07-26T06:49:13.862 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 450971566082 2021-07-26T06:49:13.862 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509483 3-450971566083 4-107374182442 5-450971566082' 2021-07-26T06:49:13.862 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.862 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:49:13.862 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855400 2021-07-26T06:49:13.863 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855400 2021-07-26T06:49:13.863 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509483 3-450971566083 4-107374182442 5-450971566082 6-150323855400' 2021-07-26T06:49:13.863 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.863 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:49:13.864 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691878 2021-07-26T06:49:13.864 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691878 2021-07-26T06:49:13.864 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509483 3-450971566083 4-107374182442 5-450971566082 6-150323855400 7-171798691878' 2021-07-26T06:49:13.864 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.864 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:49:13.865 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528358 2021-07-26T06:49:13.865 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528358 2021-07-26T06:49:13.865 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509483 3-450971566083 4-107374182442 5-450971566082 6-150323855400 7-171798691878 8-193273528358' 2021-07-26T06:49:13.865 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:13.865 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:49:13.866 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=420906795016 2021-07-26T06:49:14.872 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -ntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 420906795015 2021-07-26T06:49:14.872 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673005 2-64424509484 3-450971566082 4-107374182441 5-450971566083 6-150323855399 7-171798691879 8-193273528357 9-420906795015' 2021-07-26T06:49:14.872 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:14.873 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836524 2021-07-26T06:49:14.873 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:14.873 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:49:14.873 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836524 2021-07-26T06:49:14.874 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:14.874 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836524 2021-07-26T06:49:14.874 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2021-07-26T06:49:14.874 INFO:tasks.workunit.client.0.smithi071.stderr:251997: waiting osd.0 seq 21474836524 2021-07-26T06:49:14.875 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:49:14.875 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836525 -lt 21474836524 2021-07-26T06:49:14.875 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:14.875 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673005 2021-07-26T06:49:14.876 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:14.876 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:49:14.876 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673005 2021-07-26T06:49:14.876 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:14.876 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673005 2021-07-26T06:49:14.877 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673005' 2021-07-26T06:49:14.877 INFO:tasks.workunit.client.0.smithi071.stderr:251997: waiting osd.1 seq 42949673005 2021-07-26T06:49:14.877 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:49:14.877 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673006 -lt 42949673005 2021-07-26T06:49:14.878 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:14.878 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509484 2021-07-26T06:49:14.878 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:14.878 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:49:14.878 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509484 2021-07-26T06:49:14.879 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:14.879 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509484 2021-07-26T06:49:14.879 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2021-07-26T06:49:14.879 INFO:tasks.workunit.client.0.smithi071.stderr:251997: waiting osd.2 seq 64424509484 2021-07-26T06:49:14.880 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:49:14.880 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509484 -lt 64424509484 2021-07-26T06:49:14.880 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:208z 420906795016 2021-07-26T06:49:14.881 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509483 3-450971566083 4-107374182442 5-450971566082 6-150323855400 7-171798691878 8-193273528358 9-420906795016' 2021-07-26T06:49:14.881 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:14.881 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836525 2021-07-26T06:49:14.882 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:14.882 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:49:14.882 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836525 2021-07-26T06:49:14.882 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:14.883 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836525 2021-07-26T06:49:14.883 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2021-07-26T06:49:14.883 INFO:tasks.workunit.client.0.smithi071.stderr:251995: waiting osd.0 seq 21474836525 2021-07-26T06:49:14.883 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:49:14.884 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836525 -lt 21474836525 2021-07-26T06:49:14.884 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:14.884 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673004 2021-07-26T06:49:14.884 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:14.885 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:49:14.885 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673004 2021-07-26T06:49:14.885 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:14.885 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673004 2021-07-26T06:49:14.885 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2021-07-26T06:49:14.886 INFO:tasks.workunit.client.0.smithi071.stderr:251995: waiting osd.1 seq 42949673004 2021-07-26T06:49:14.886 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:49:14.886 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673006 -lt 42949673004 2021-07-26T06:49:14.886 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:14.887 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509483 2021-07-26T06:49:14.887 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:14.887 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:49:14.887 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509483 2021-07-26T06:49:14.888 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:14.888 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509483 2021-07-26T06:49:14.888 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509483' 2021-07-26T06:49:14.888 INFO:tasks.workunit.client.0.smithi071.stderr:251995: waiting osd.2 seq 64424509483 2021-07-26T06:49:14.889 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:49:14.889 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509484 -lt 64424509483 2021-07-26T06:49:14.889 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:15.761 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-450971566083 2021-07-26T06:49:15.761 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:15.761 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:49:15.762 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-450971566083 2021-07-26T06:49:15.762 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:15.762 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=450971566083 2021-07-26T06:49:15.762 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 450971566083' 2021-07-26T06:49:15.763 INFO:tasks.workunit.client.0.smithi071.stderr:251995: waiting osd.3 seq 450971566083 2021-07-26T06:49:15.763 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:49:15.763 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 450971566083 -lt 450971566083 2021-07-26T06:49:15.763 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:15.764 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182442 2021-07-26T06:49:15.764 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:15.764 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:49:15.764 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182442 2021-07-26T06:49:15.764 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:15.765 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182442 2021-07-26T06:49:15.765 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2021-07-26T06:49:15.765 INFO:tasks.workunit.client.0.smithi071.stderr:251995: waiting osd.4 seq 107374182442 2021-07-26T06:49:15.765 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:49:15.766 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182442 -lt 107374182442 2021-07-26T06:49:15.766 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:15.766 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-450971566082 2021-07-26T06:49:15.766 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:15.767 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:49:15.767 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-450971566082 2021-07-26T06:49:15.767 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:15.767 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=450971566082 2021-07-26T06:49:15.767 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 450971566082' 2021-07-26T06:49:15.768 INFO:tasks.workunit.client.0.smithi071.stderr:251995: waiting osd.5 seq 450971566082 2021-07-26T06:49:15.768 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:49:15.768 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 450971566084 -lt 450971566082 2021-07-26T06:49:15.768 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:15.768 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855400 2021-07-26T06:49:15.769 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:15.769 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:49:15.769 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855400 2021-07-26T06:49:15.772 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /4: flush_pg_stats: for s in $seqs 2021-07-26T06:49:15.772 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-450971566082 2021-07-26T06:49:15.773 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:15.773 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:49:15.773 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-450971566082 2021-07-26T06:49:15.773 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:15.774 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=450971566082 2021-07-26T06:49:15.774 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 450971566082' 2021-07-26T06:49:15.774 INFO:tasks.workunit.client.0.smithi071.stderr:251997: waiting osd.3 seq 450971566082 2021-07-26T06:49:15.774 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:49:15.774 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 450971566083 -lt 450971566082 2021-07-26T06:49:15.775 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:15.775 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182441 2021-07-26T06:49:15.775 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:15.775 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:49:15.776 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182441 2021-07-26T06:49:15.776 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:15.776 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182441 2021-07-26T06:49:15.776 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182441' 2021-07-26T06:49:15.776 INFO:tasks.workunit.client.0.smithi071.stderr:251997: waiting osd.4 seq 107374182441 2021-07-26T06:49:15.777 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:49:15.777 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182442 -lt 107374182441 2021-07-26T06:49:15.777 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:15.777 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-450971566083 2021-07-26T06:49:15.778 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:15.778 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:49:15.778 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-450971566083 2021-07-26T06:49:15.778 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:15.778 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=450971566083 2021-07-26T06:49:15.779 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 450971566083' 2021-07-26T06:49:15.779 INFO:tasks.workunit.client.0.smithi071.stderr:251997: waiting osd.5 seq 450971566083 2021-07-26T06:49:15.779 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:49:15.779 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 450971566084 -lt 450971566083 2021-07-26T06:49:15.780 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:15.780 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855399 2021-07-26T06:49:15.780 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:15.780 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:49:16.782 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855399 2021-07-26T06:49:16.782 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:16.782 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855399 2021-07-26T06:49:16.782 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855399' 2021-07-26T06:49:16.783 INFO:tasks.workunit.client.0.smithi071.stderr:251997: waiting osd.6 seq 150323855399 2021-07-26T06:49:16.783 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:49:16.783 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855400 -lt 150323855399 2021-07-26T06:49:16.783 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:16.784 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691879 2021-07-26T06:49:16.784 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:16.784 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:49:16.784 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691879 2021-07-26T06:49:16.784 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:16.785 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691879 2021-07-26T06:49:16.785 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2021-07-26T06:49:16.785 INFO:tasks.workunit.client.0.smithi071.stderr:251997: waiting osd.7 seq 171798691879 2021-07-26T06:49:16.785 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:49:16.785 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691879 -lt 171798691879 2021-07-26T06:49:16.786 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:16.786 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528357 2021-07-26T06:49:16.786 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:16.786 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:49:16.787 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528357 2021-07-26T06:49:16.787 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:16.787 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528357 2021-07-26T06:49:16.787 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528357' 2021-07-26T06:49:16.787 INFO:tasks.workunit.client.0.smithi071.stderr:251997: waiting osd.8 seq 193273528357 2021-07-26T06:49:16.788 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:49:16.788 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528358 -lt 193273528357 2021-07-26T06:49:16.788 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:16.788 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-420906795015 2021-07-26T06:49:16.789 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:16.789 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:49:16.789 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-420906795015 2021-07-26T06:49:16.789 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:16.789 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=420906795015 2021-07-26T06:49:16.790 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'wa/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:16.790 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855400 2021-07-26T06:49:16.790 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855400' 2021-07-26T06:49:16.791 INFO:tasks.workunit.client.0.smithi071.stderr:251995: waiting osd.6 seq 150323855400 2021-07-26T06:49:16.791 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:49:16.791 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855400 -lt 150323855400 2021-07-26T06:49:16.791 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:16.792 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691878 2021-07-26T06:49:16.792 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:16.792 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:49:16.792 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691878 2021-07-26T06:49:16.793 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:16.793 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691878 2021-07-26T06:49:16.793 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691878' 2021-07-26T06:49:16.793 INFO:tasks.workunit.client.0.smithi071.stderr:251995: waiting osd.7 seq 171798691878 2021-07-26T06:49:16.793 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:49:16.794 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691879 -lt 171798691878 2021-07-26T06:49:16.794 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:16.794 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528358 2021-07-26T06:49:16.794 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:16.795 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:49:16.795 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528358 2021-07-26T06:49:16.795 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:16.795 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528358 2021-07-26T06:49:16.795 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2021-07-26T06:49:16.796 INFO:tasks.workunit.client.0.smithi071.stderr:251995: waiting osd.8 seq 193273528358 2021-07-26T06:49:16.796 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:49:16.796 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528358 -lt 193273528358 2021-07-26T06:49:16.796 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:16.796 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-420906795016 2021-07-26T06:49:16.797 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:16.797 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:49:16.797 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-420906795016 2021-07-26T06:49:16.797 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:16.798 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=420906795016 2021-07-26T06:49:16.798 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 420906795016' 2021-07-26T06:49:16.798 INFO:tasks.workunit.client.0.smithi071.stderr:251995: waiting osd.9 seq 420906795016 2021-07-26T06:49:18.045 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuiting osd.9 seq 420906795015' 2021-07-26T06:49:18.046 INFO:tasks.workunit.client.0.smithi071.stderr:251997: waiting osd.9 seq 420906795015 2021-07-26T06:49:18.046 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:49:18.046 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 420906795016 -lt 420906795015 2021-07-26T06:49:18.046 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:49:18.047 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:18.047 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:18.047 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:49:18.047 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:49:18.047 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:49:18.048 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:49:18.048 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:49:18.048 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:49:18.049 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:49:18.049 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:49:18.049 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:49:18.050 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:49:18.050 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:18.050 INFO:tasks.workunit.client.0.smithi071.stderr:251997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:18.051 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:49:18.051 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:49:18.051 INFO:tasks.workunit.client.0.smithi071.stderr:251997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:49:18.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:49:18.271 INFO:tasks.workunit.client.0.smithi071.stderr:ntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:49:18.272 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 420906795016 -lt 420906795016 2021-07-26T06:49:18.272 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:49:18.272 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:18.272 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:18.273 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:49:18.273 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:49:18.273 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:49:18.273 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:49:18.274 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:49:18.274 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:49:18.274 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:49:18.274 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:49:18.274 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:49:18.275 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:49:18.275 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:18.275 INFO:tasks.workunit.client.0.smithi071.stderr:251995: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:18.275 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:49:18.276 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:49:18.276 INFO:tasks.workunit.client.0.smithi071.stderr:251995: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:49:18.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:49:18.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:49:18.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 251994 2021-07-26T06:49:18.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:49:18.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:49:18.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:49:18.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2021-07-26T06:49:18.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:49:18.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2021-07-26T06:49:18.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:49:18.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:49:18.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:49:18.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:49:18.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2021-07-26T06:49:18.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2021-07-26T06:49:18.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:49:18.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:49:18.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:49:18.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:49:18.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:49:18.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:49:18.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:48:53.680531+0000 2021-07-26T06:49:18.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:49:19.091 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:49:19.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:48:53.680531+0000 2021-07-26T06:49:19.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:49:19.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:48:53.680531+0000 2021-07-26T06:49:19.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:49:19.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:49:19.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:49:19.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:49:19.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:49:19.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:49:19.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:49:19.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:49:19.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:48:53.680531+0000 '>' 2021-07-26T06:48:53.680531+0000 2021-07-26T06:49:19.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:49:20.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:49:20.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:49:20.367 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:49:20.367 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:49:20.367 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:49:20.368 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:49:20.368 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:49:20.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:48:53.680531+0000 '>' 2021-07-26T06:48:53.680531+0000 2021-07-26T06:49:20.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:49:21.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:49:21.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:49:21.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:49:21.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:49:21.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:49:21.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:49:21.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:49:21.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:49:19.525160+0000 '>' 2021-07-26T06:48:53.680531+0000 2021-07-26T06:49:21.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:49:21.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2021-07-26T06:49:21.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:49:21.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:49:21.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:49:21.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 254044"' 2021-07-26T06:49:21.956 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 254044' 2021-07-26T06:49:21.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:49:21.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:49:21.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:49:21.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 254046"' 2021-07-26T06:49:21.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:49:21.957 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 254046' 2021-07-26T06:49:21.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2021-07-26T06:49:21.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 254044 254046' 2021-07-26T06:49:21.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/254047: /' 2021-07-26T06:49:21.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:49:21.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:49:21.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 254044 2021-07-26T06:49:21.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:49:21.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/254049: /' 2021-07-26T06:49:23.857 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:49:23.858 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:49:23.858 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=5 2021-07-26T06:49:23.858 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:49:23.858 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:49:23.859 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:49:23.859 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:49:23.859 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=5 2021-07-26T06:49:23.859 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:49:23.860 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2021-07-26T06:49:23.860 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:49:23.860 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:49:23.860 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:49:23.861 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:49:23.861 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:49:23.861 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:49:23.861 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:49:23.861 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:49:23.862 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:49:23.862 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=5 2021-07-26T06:49:23.862 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:49:23.862 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:49:23.863 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2021-07-26T06:49:23.863 INFO:tasks.workunit.client.0.smithi071.stderr:254049: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:49:23.863 INFO:tasks.workunit.client.0.smithi071.stderr:254049: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:49:23.864 INFO:tasks.workunit.client.0.smithi071.stderr:254049: _ 2021-07-26T06:49:23.864 INFO:tasks.workunit.client.0.smithi071.stderr:254049: hinfo_key 2021-07-26T06:49:23.864 INFO:tasks.workunit.client.0.smithi071.stderr:254049: snapset 2021-07-26T06:49:23.864 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2021-07-26T06:49:23.865 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:49:23.865 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:49:23.865 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=5 2021-07-26T06:49:23.865 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:49:23.865 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:49:23.866 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:49:23.866 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clon254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:49:23.866 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:49:23.866 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:49:23.867 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:49:23.867 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:49:23.867 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:49:23.867 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:49:23.867 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:49:23.868 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:49:23.868 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:49:23.868 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:49:23.868 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:49:23.868 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:49:23.869 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:49:23.869 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:49:23.869 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:49:23.869 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:49:23.870 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:49:23.870 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:49:23.871 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:49:23.871 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:49:23.871 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:49:23.871 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2021-07-26T06:49:23.872 INFO:tasks.workunit.client.0.smithi071.stderr:254047: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2021-07-26T06:49:23.872 INFO:tasks.workunit.client.0.smithi071.stderr:254047: Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:49:23.872 INFO:tasks.workunit.client.0.smithi071.stderr:254047: _ 2021-07-26T06:49:23.872 INFO:tasks.workunit.client.0.smithi071.stderr:254047: hinfo_key 2021-07-26T06:49:23.872 INFO:tasks.workunit.client.0.smithi071.stderr:254047: snapset 2021-07-26T06:49:23.873 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:49:23.873 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:49:23.873 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:49:23.873 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:49:23.874 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:49:23.874 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:49:23.874 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:49:23.875 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubunte.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:49:23.875 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:49:23.875 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:49:23.876 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2021-07-26T06:49:23.876 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2021-07-26T06:49:23.876 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:49:23.877 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:49:23.877 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:49:23.877 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:49:23.877 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:49:23.878 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:49:23.878 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:49:23.878 INFO:tasks.workunit.client.0.smithi071.stderr:254049: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:49:23.879 INFO:tasks.workunit.client.0.smithi071.stderr:254049: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:49:23.879 INFO:tasks.workunit.client.0.smithi071.stderr:254049: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:49:23.879 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:23.879 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:23.879 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:49:23.880 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:49:23.880 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:49:23.880 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:49:23.880 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:49:23.881 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:49:23.881 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:49:23.881 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:49:23.881 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/5 2021-07-26T06:49:23.881 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.5 2021-07-26T06:49:23.882 INFO:tasks.workunit.client.0.smithi071.stderr:254049: start osd.5 2021-07-26T06:49:23.882 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scruu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:49:23.882 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:49:23.882 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:49:23.883 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:49:23.883 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:49:23.883 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:49:23.883 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:49:23.884 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:49:23.884 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:49:23.884 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:49:23.885 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:49:23.885 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:49:23.885 INFO:tasks.workunit.client.0.smithi071.stderr:254047: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:49:23.885 INFO:tasks.workunit.client.0.smithi071.stderr:254047: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:49:23.885 INFO:tasks.workunit.client.0.smithi071.stderr:254047: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:49:23.886 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:23.886 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:23.886 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:49:23.886 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:49:23.887 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:49:23.887 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:49:23.887 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:49:23.887 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:49:23.888 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:49:23.888 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:49:23.888 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:49:23.888 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:49:23.888 INFO:tasks.workunit.client.0.smithi071.stderr:254047: start osd.3 2021-07-26T06:49:25.841 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:49:25.842 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:49:25.843 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:49:25.843 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:49:25.843 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:49:25.844 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:49:25.844 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 2021-07-26T06:49:23.894+0000 7f96f1d9a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:25.845 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 2021-07-26T06:49:23.910+0000 7f96f1d9a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:25.845 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 2021-07-26T06:49:23.911+0000 7f96f1d9a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:25.846 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 2021-07-26T06:49:23.913+0000 7f96f1d9a0c0 -1 Falling back to public interface 2021-07-26T06:49:25.846 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:49:25.847 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:49:25.847 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:49:25.847 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:49:25.848 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:49:25.848 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:25.849 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:49:25.849 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 0 2021-07-26T06:49:25.849 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:25.850 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:49:25.850 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:49:25.851 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 2021-07-26T06:49:24.832+0000 7f96f1d9a0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:49:25.851 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 2021-07-26T06:49:24.840+0000 7f96f1d9a0c0 -1 journal do_read_entry(1150976): bad header magic 2021-07-26T06:49:25.852 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 2021-07-26T06:49:24.840+0000 7f96f1d9a0c0 -1 journal do_read_entry(1150976): bad header magic 2021-07-26T06:49:25.852 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 2021-07-26T06:49:24.847+0000 7f96f1d9a0c0 -1 osd.3 107 log_to_monitors {default=true} 2021-07-26T06:49:25.852 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:49:25.853 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:25.853 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:49:25.854 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 1 2021-07-26T06:49:25.854 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:25.855 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:49:25.855 INFO:tasks.workunit.client.0.smithi071.stderr:254047: osd.3 up in weight 1 up_from 111 up_thru 111 down_at 108 last_clean_interval [105,107) [v2:127.0.0.1:6827/254191,v1:127.0.0.1:6830/254191] [v2:127.0.0.1:6833/254191,v1:127.0.0.1:6845/254191] exists,up ae2ab4fe-2368-4e96-8e81-aad31b7d34ea 2021-07-26T06:49:25.855 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:49:25.856 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:49:25.856 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:49:25.857 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:49:25.857 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:49:25.858 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.clib-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:49:25.858 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/5/whoami 2021-07-26T06:49:25.859 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 5 = 5 ']' 2021-07-26T06:49:25.859 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:49:25.859 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:49:25.860 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:49:25.860 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 2021-07-26T06:49:23.884+0000 7fa932fe00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:25.861 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 2021-07-26T06:49:23.910+0000 7fa932fe00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:25.861 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 2021-07-26T06:49:23.911+0000 7fa932fe00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:25.862 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 2021-07-26T06:49:23.913+0000 7fa932fe00c0 -1 Falling back to public interface 2021-07-26T06:49:25.862 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 5 2021-07-26T06:49:25.862 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:49:25.863 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:49:25.863 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:49:25.863 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:49:25.864 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:25.864 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:49:25.864 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 0 2021-07-26T06:49:25.865 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:25.865 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:49:25.865 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:49:25.866 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 2021-07-26T06:49:24.832+0000 7fa932fe00c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:49:25.866 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 2021-07-26T06:49:24.840+0000 7fa932fe00c0 -1 journal do_read_entry(1187840): bad header magic 2021-07-26T06:49:25.867 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 2021-07-26T06:49:24.840+0000 7fa932fe00c0 -1 journal do_read_entry(1187840): bad header magic 2021-07-26T06:49:25.867 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 2021-07-26T06:49:24.847+0000 7fa932fe00c0 -1 osd.5 107 log_to_monitors {default=true} 2021-07-26T06:49:25.867 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:49:25.867 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:25.867 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:49:25.868 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 1 2021-07-26T06:49:25.868 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:25.868 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:49:25.868 INFO:tasks.workunit.client.0.smithi071.stderr:254049: osd.5 up in weight 1 up_from 111 up_thru 111 down_at 108 last_clean_interval [105,107) [v2:127.0.0.1:6826/254190,v1:127.0.0.1:6828/254190] [v2:127.0.0.1:6829/254190,v1:127.0.0.1:6831/254190] exists,up 10f94d47-7b45-43ca-bed4-f3a0dac66239 2021-07-26T06:49:25.869 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:49:25.869 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:49:25.869 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:49:25.869 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:49:25.870 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:49:26.928 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:49:26.928 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:49:26.929 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:49:26.929 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:49:26.929 INFO:tasks.workunit.client.0.smithi071.stderr:254049: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:49:26.929 INFO:tasks.workunit.client.0.smithi071.stderr:254049: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:49:26.930 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:49:26.930 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:49:26.930 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:49:26.930 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:49:26.931 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:49:26.931 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:49:26.931 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:49:26.931 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:49:26.932 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:49:26.932 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 1 2021-07-26T06:49:26.932 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 2 2021-07-26T06:49:26.932 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 3 2021-07-26T06:49:26.932 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 4 2021-07-26T06:49:26.933 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 5 2021-07-26T06:49:26.933 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 6 2021-07-26T06:49:26.933 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 7 2021-07-26T06:49:26.933 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 8 2021-07-26T06:49:26.933 INFO:tasks.workunit.client.0.smithi071.stderr:254049: 9' 2021-07-26T06:49:26.934 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:49:26.934 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:26.934 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:49:26.934 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836529 2021-07-26T06:49:26.935 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836529 2021-07-26T06:49:26.935 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529' 2021-07-26T06:49:26.935 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:26.935 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:49:26.935 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673009 2021-07-26T06:49:26.936 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673009 2021-07-26T06:49:26.936 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673009' 2021-07-26T06:49:26.936 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:26.936 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:49:26.937 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509487 2021-07-26T06:49:26.937 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509487 2021-07-26T06:49:26.937 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673009 2-64424509487' 2021-07-26T06:49:26.937 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:26.938 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:49:26.938 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats:ent.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:49:26.938 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:49:26.938 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:49:26.938 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:49:26.939 INFO:tasks.workunit.client.0.smithi071.stderr:254047: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:49:26.939 INFO:tasks.workunit.client.0.smithi071.stderr:254047: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:49:26.939 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:49:26.939 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:49:26.939 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:49:26.940 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:49:26.940 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:49:26.940 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:49:26.940 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:49:26.940 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:49:26.941 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:49:26.941 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 1 2021-07-26T06:49:26.941 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 2 2021-07-26T06:49:26.941 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 3 2021-07-26T06:49:26.941 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 4 2021-07-26T06:49:26.942 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 5 2021-07-26T06:49:26.942 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 6 2021-07-26T06:49:26.942 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 7 2021-07-26T06:49:26.942 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 8 2021-07-26T06:49:26.942 INFO:tasks.workunit.client.0.smithi071.stderr:254047: 9' 2021-07-26T06:49:26.943 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:49:26.943 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:26.943 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:49:26.943 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836530 2021-07-26T06:49:26.943 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836530 2021-07-26T06:49:26.944 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836530' 2021-07-26T06:49:26.944 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:26.944 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:49:26.944 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673010 2021-07-26T06:49:26.945 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673010 2021-07-26T06:49:26.945 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836530 1-42949673010' 2021-07-26T06:49:26.945 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:26.945 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:49:26.946 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509488 2021-07-26T06:49:26.946 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509488 2021-07-26T06:49:26.946 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836530 1-42949673010 2-64424509488' 2021-07-26T06:49:26.946 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:26.946 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:49:27.686 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=476741369859 2021-07-26T06:49:27.687 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 476741369859 2021-07-26T06:49:27.687 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836530 1-42949673010 2-64424509488 3-476741369859' 2021-07-26T06:49:27.688 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:27.688 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:49:27.688 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182447 2021-07-26T06:49:27.689 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182447 2021-07-26T06:49:27.689 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836530 1-42949673010 2-64424509488 3-476741369859 4-107374182447' 2021-07-26T06:49:27.689 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:27.689 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:49:27.690 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=476741369859 2021-07-26T06:49:27.690 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 476741369859 2021-07-26T06:49:27.690 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836530 1-42949673010 2-64424509488 3-476741369859 4-107374182447 5-476741369859' 2021-07-26T06:49:27.690 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:27.691 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:49:27.691 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855404 2021-07-26T06:49:27.691 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855404 2021-07-26T06:49:27.691 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836530 1-42949673010 2-64424509488 3-476741369859 4-107374182447 5-476741369859 6-150323855404' 2021-07-26T06:49:27.691 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:27.692 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:49:27.692 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691883 2021-07-26T06:49:27.692 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691883 2021-07-26T06:49:27.692 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836530 1-42949673010 2-64424509488 3-476741369859 4-107374182447 5-476741369859 6-150323855404 7-171798691883' 2021-07-26T06:49:27.693 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:27.693 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:49:27.693 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528361 2021-07-26T06:49:27.693 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528361 2021-07-26T06:49:27.694 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836530 1-42949673010 2-64424509488 3-476741369859 4-107374182447 5-476741369859 6-150323855404 7-171798691883 8-193273528361' 2021-07-26T06:49:27.694 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:27.694 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:49:27.694 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=4209 seq=476741369858 2021-07-26T06:49:27.695 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 476741369858 2021-07-26T06:49:27.695 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673009 2-64424509487 3-476741369858' 2021-07-26T06:49:27.695 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:27.696 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:49:27.696 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182446 2021-07-26T06:49:27.697 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182446 2021-07-26T06:49:27.697 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673009 2-64424509487 3-476741369858 4-107374182446' 2021-07-26T06:49:27.698 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:27.698 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:49:27.698 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=476741369858 2021-07-26T06:49:27.699 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 476741369858 2021-07-26T06:49:27.699 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673009 2-64424509487 3-476741369858 4-107374182446 5-476741369858' 2021-07-26T06:49:27.700 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:27.700 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:49:27.701 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855405 2021-07-26T06:49:27.701 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855405 2021-07-26T06:49:27.701 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673009 2-64424509487 3-476741369858 4-107374182446 5-476741369858 6-150323855405' 2021-07-26T06:49:27.702 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:27.702 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:49:27.702 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691882 2021-07-26T06:49:27.703 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691882 2021-07-26T06:49:27.703 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673009 2-64424509487 3-476741369858 4-107374182446 5-476741369858 6-150323855405 7-171798691882' 2021-07-26T06:49:27.704 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:27.704 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:49:27.704 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528362 2021-07-26T06:49:27.705 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528362 2021-07-26T06:49:27.705 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673009 2-64424509487 3-476741369858 4-107374182446 5-476741369858 6-150323855405 7-171798691882 8-193273528362' 2021-07-26T06:49:27.705 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:27.705 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:49:27.706 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=420906795020 2021-07-26T06:49:28.626 INFO:tasks.workunit.client.0.smithi071.stderr:25404906795019 2021-07-26T06:49:28.626 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 420906795019 2021-07-26T06:49:28.626 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836530 1-42949673010 2-64424509488 3-476741369859 4-107374182447 5-476741369859 6-150323855404 7-171798691883 8-193273528361 9-420906795019' 2021-07-26T06:49:28.627 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:28.627 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836530 2021-07-26T06:49:28.628 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:28.628 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:49:28.628 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836530 2021-07-26T06:49:28.628 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:28.628 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836530 2021-07-26T06:49:28.629 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2021-07-26T06:49:28.629 INFO:tasks.workunit.client.0.smithi071.stderr:254047: waiting osd.0 seq 21474836530 2021-07-26T06:49:28.629 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:49:28.629 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836530 -lt 21474836530 2021-07-26T06:49:28.630 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:28.630 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673010 2021-07-26T06:49:28.630 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:28.630 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:49:28.631 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673010 2021-07-26T06:49:28.631 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:28.631 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673010 2021-07-26T06:49:28.631 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2021-07-26T06:49:28.632 INFO:tasks.workunit.client.0.smithi071.stderr:254047: waiting osd.1 seq 42949673010 2021-07-26T06:49:28.632 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:49:28.632 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673010 -lt 42949673010 2021-07-26T06:49:28.632 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:28.632 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509488 2021-07-26T06:49:28.633 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:28.633 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:49:28.633 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509488 2021-07-26T06:49:28.633 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:28.634 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509488 2021-07-26T06:49:28.634 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509488' 2021-07-26T06:49:28.634 INFO:tasks.workunit.client.0.smithi071.stderr:254047: waiting osd.2 seq 64424509488 2021-07-26T06:49:28.634 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:49:28.634 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509489 -lt 64424509488 2021-07-26T06:49:28.635 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/stan: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 420906795020 2021-07-26T06:49:28.635 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673009 2-64424509487 3-476741369858 4-107374182446 5-476741369858 6-150323855405 7-171798691882 8-193273528362 9-420906795020' 2021-07-26T06:49:28.636 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:28.636 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836529 2021-07-26T06:49:28.636 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:28.636 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:49:28.636 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836529 2021-07-26T06:49:28.637 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:28.637 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836529 2021-07-26T06:49:28.637 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2021-07-26T06:49:28.637 INFO:tasks.workunit.client.0.smithi071.stderr:254049: waiting osd.0 seq 21474836529 2021-07-26T06:49:28.638 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:49:28.638 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836530 -lt 21474836529 2021-07-26T06:49:28.638 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:28.638 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673009 2021-07-26T06:49:28.638 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:28.639 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:49:28.639 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673009 2021-07-26T06:49:28.639 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:28.639 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673009 2021-07-26T06:49:28.640 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673009' 2021-07-26T06:49:28.640 INFO:tasks.workunit.client.0.smithi071.stderr:254049: waiting osd.1 seq 42949673009 2021-07-26T06:49:28.640 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:49:28.640 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673010 -lt 42949673009 2021-07-26T06:49:28.640 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:28.641 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509487 2021-07-26T06:49:28.641 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:28.641 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:49:28.641 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509487 2021-07-26T06:49:28.642 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:28.642 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509487 2021-07-26T06:49:28.642 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509487' 2021-07-26T06:49:28.642 INFO:tasks.workunit.client.0.smithi071.stderr:254049: waiting osd.2 seq 64424509487 2021-07-26T06:49:28.642 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:49:28.643 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509489 -lt 64424509487 2021-07-26T06:49:29.567 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:29.567 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-476741369858 2021-07-26T06:49:29.567 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:29.568 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:49:29.568 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-476741369858 2021-07-26T06:49:29.568 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:29.568 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=476741369858 2021-07-26T06:49:29.569 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 476741369858' 2021-07-26T06:49:29.569 INFO:tasks.workunit.client.0.smithi071.stderr:254049: waiting osd.3 seq 476741369858 2021-07-26T06:49:29.569 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:49:29.569 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 476741369859 -lt 476741369858 2021-07-26T06:49:29.569 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:29.570 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182446 2021-07-26T06:49:29.570 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:29.570 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:49:29.570 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182446 2021-07-26T06:49:29.571 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:29.571 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182446 2021-07-26T06:49:29.571 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2021-07-26T06:49:29.571 INFO:tasks.workunit.client.0.smithi071.stderr:254049: waiting osd.4 seq 107374182446 2021-07-26T06:49:29.571 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:49:29.572 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182447 -lt 107374182446 2021-07-26T06:49:29.572 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:29.572 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-476741369858 2021-07-26T06:49:29.572 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:29.572 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:49:29.573 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-476741369858 2021-07-26T06:49:29.573 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:29.573 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=476741369858 2021-07-26T06:49:29.573 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 476741369858' 2021-07-26T06:49:29.574 INFO:tasks.workunit.client.0.smithi071.stderr:254049: waiting osd.5 seq 476741369858 2021-07-26T06:49:29.574 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:49:29.574 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 476741369859 -lt 476741369858 2021-07-26T06:49:29.574 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:29.574 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855405 2021-07-26T06:49:29.575 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:29.575 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:49:29.575 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtestdalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:29.576 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-476741369859 2021-07-26T06:49:29.576 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:29.576 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:49:29.576 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-476741369859 2021-07-26T06:49:29.577 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:29.577 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=476741369859 2021-07-26T06:49:29.577 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 476741369859' 2021-07-26T06:49:29.577 INFO:tasks.workunit.client.0.smithi071.stderr:254047: waiting osd.3 seq 476741369859 2021-07-26T06:49:29.578 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:49:29.578 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 476741369859 -lt 476741369859 2021-07-26T06:49:29.578 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:29.578 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182447 2021-07-26T06:49:29.578 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:29.579 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:49:29.579 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182447 2021-07-26T06:49:29.579 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:29.579 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182447 2021-07-26T06:49:29.579 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182447' 2021-07-26T06:49:29.580 INFO:tasks.workunit.client.0.smithi071.stderr:254047: waiting osd.4 seq 107374182447 2021-07-26T06:49:29.580 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:49:29.580 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182447 -lt 107374182447 2021-07-26T06:49:29.580 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:29.581 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-476741369859 2021-07-26T06:49:29.581 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:29.581 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:49:29.581 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-476741369859 2021-07-26T06:49:29.581 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:29.582 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=476741369859 2021-07-26T06:49:29.582 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 476741369859' 2021-07-26T06:49:29.582 INFO:tasks.workunit.client.0.smithi071.stderr:254047: waiting osd.5 seq 476741369859 2021-07-26T06:49:29.582 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:49:29.583 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 476741369859 -lt 476741369859 2021-07-26T06:49:29.583 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:29.583 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855404 2021-07-26T06:49:29.583 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:29.583 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:49:30.540 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855404 2021-07-26T06:49:30.540 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:30.540 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855404 2021-07-26T06:49:30.540 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855404' 2021-07-26T06:49:30.541 INFO:tasks.workunit.client.0.smithi071.stderr:254047: waiting osd.6 seq 150323855404 2021-07-26T06:49:30.541 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:49:30.541 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855405 -lt 150323855404 2021-07-26T06:49:30.541 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:30.542 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691883 2021-07-26T06:49:30.542 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:30.542 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:49:30.542 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691883 2021-07-26T06:49:30.543 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:30.543 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691883 2021-07-26T06:49:30.543 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2021-07-26T06:49:30.543 INFO:tasks.workunit.client.0.smithi071.stderr:254047: waiting osd.7 seq 171798691883 2021-07-26T06:49:30.544 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:49:30.544 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691884 -lt 171798691883 2021-07-26T06:49:30.544 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:30.544 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528361 2021-07-26T06:49:30.544 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:30.545 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:49:30.545 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528361 2021-07-26T06:49:30.545 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:30.545 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528361 2021-07-26T06:49:30.546 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2021-07-26T06:49:30.546 INFO:tasks.workunit.client.0.smithi071.stderr:254047: waiting osd.8 seq 193273528361 2021-07-26T06:49:30.546 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:49:30.546 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528363 -lt 193273528361 2021-07-26T06:49:30.546 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:30.547 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-420906795019 2021-07-26T06:49:30.547 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:30.547 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:49:30.547 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-420906795019 2021-07-26T06:49:30.548 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:30.548 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=420906795019 2021-07-26T06:49:30.548 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087:/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855405 2021-07-26T06:49:30.549 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:30.549 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855405 2021-07-26T06:49:30.549 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855405' 2021-07-26T06:49:30.549 INFO:tasks.workunit.client.0.smithi071.stderr:254049: waiting osd.6 seq 150323855405 2021-07-26T06:49:30.550 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:49:30.550 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855405 -lt 150323855405 2021-07-26T06:49:30.550 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:30.550 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691882 2021-07-26T06:49:30.551 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:30.551 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:49:30.551 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691882 2021-07-26T06:49:30.551 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:30.551 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691882 2021-07-26T06:49:30.552 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691882' 2021-07-26T06:49:30.552 INFO:tasks.workunit.client.0.smithi071.stderr:254049: waiting osd.7 seq 171798691882 2021-07-26T06:49:30.552 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:49:30.553 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691884 -lt 171798691882 2021-07-26T06:49:30.553 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:30.553 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528362 2021-07-26T06:49:30.553 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:30.553 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:49:30.554 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528362 2021-07-26T06:49:30.554 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:30.554 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528362 2021-07-26T06:49:30.554 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528362' 2021-07-26T06:49:30.554 INFO:tasks.workunit.client.0.smithi071.stderr:254049: waiting osd.8 seq 193273528362 2021-07-26T06:49:30.555 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:49:30.555 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528363 -lt 193273528362 2021-07-26T06:49:30.555 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:30.555 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-420906795020 2021-07-26T06:49:30.556 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:30.556 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:49:30.556 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-420906795020 2021-07-26T06:49:30.556 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:30.556 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=420906795020 2021-07-26T06:49:31.875 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats flush_pg_stats: echo 'waiting osd.9 seq 420906795019' 2021-07-26T06:49:31.876 INFO:tasks.workunit.client.0.smithi071.stderr:254047: waiting osd.9 seq 420906795019 2021-07-26T06:49:31.876 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:49:31.876 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 420906795021 -lt 420906795019 2021-07-26T06:49:31.876 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:49:31.877 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:31.877 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:31.877 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:49:31.877 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:49:31.878 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:49:31.878 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:49:31.878 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:49:31.878 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:49:31.879 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:49:31.879 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:49:31.879 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:49:31.879 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:49:31.879 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:31.880 INFO:tasks.workunit.client.0.smithi071.stderr:254047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:31.880 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:49:31.880 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:49:31.880 INFO:tasks.workunit.client.0.smithi071.stderr:254047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:49:31.881 INFO:tasks.workunit.client.0.smithi071.stderr:: echo 'waiting osd.9 seq 420906795020' 2021-07-26T06:49:31.881 INFO:tasks.workunit.client.0.smithi071.stderr:254049: waiting osd.9 seq 420906795020 2021-07-26T06:49:31.881 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:49:31.881 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 420906795021 -lt 420906795020 2021-07-26T06:49:31.882 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:49:31.882 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:31.882 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:31.882 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:49:31.882 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:49:31.883 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:49:31.883 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:49:31.883 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:49:31.884 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:49:31.884 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:49:31.884 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:49:31.884 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:49:31.884 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:49:31.885 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:31.885 INFO:tasks.workunit.client.0.smithi071.stderr:254049: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:31.885 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:49:31.885 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:49:31.886 INFO:tasks.workunit.client.0.smithi071.stderr:254049: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:49:31.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:49:31.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:49:31.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:49:31.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 254046 2021-07-26T06:49:31.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:49:31.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:49:31.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:49:31.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2021-07-26T06:49:31.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:49:31.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:49:39.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:49:39.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:49:39.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:49:39.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:49:39.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:49:39.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:49:39.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:49:39.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:49:39.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:49:39.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:49:39.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:49:39.272 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:49:39.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:49:39.273 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:49:39.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:49:39.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:49:39.275 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:49:39.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:49:39.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:49:39.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:49:39.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:49:39.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:49:39.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:49:39.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:49:39.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:49:39.337 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:49:39.337 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:49:39.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:49:39.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:49:39.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:49:39.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:49:39.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:49:39.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:49:39.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:49:39.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:49:39.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:49:39.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:49:39.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:49:39.342 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:49:39.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:49:39.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:49:39.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:49:39.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:49:39.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:49:39.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:49:39.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:49:39.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:49:39.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:49:39.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:49:39.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:49:39.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:49:39.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:49:39.347 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:49:39.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:49:39.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:49:39.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:49:39.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:49:39.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:49:39.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:49:39.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:49:39.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:49:39.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:49:39.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:49:39.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:49:39.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:49:39.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:49:39.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:49:39.354 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:49:39.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:49:39.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:49:39.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:49:39.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2021-07-26T06:49:39.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:851: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2021-07-26T06:49:39.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2021-07-26T06:49:39.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2021-07-26T06:49:39.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local allow_overwrites=true 2021-07-26T06:49:39.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:828: corrupt_and_repair_lrc: local poolname=ecpool 2021-07-26T06:49:39.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2021-07-26T06:49:39.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:49:39.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:49:39.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:49:39.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:49:39.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:49:39.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T06:49:39.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:49:39.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:49:39.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:49:39.724 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:49:39.724 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:49:39.724 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:49:39.724 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:39.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:49:39.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:49:39.992 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:49:39.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:49:39.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:49:39.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:49:39.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:49:39.993 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:49:39.994 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:49:39.994 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:49:39.994 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:49:39.994 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:49:39.995 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:49:39.995 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:49:39.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:49:39.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:49:40.090 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:49:40.091 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:49:40.091 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:49:40.091 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:49:40.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:49:40.092 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:49:40.092 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:49:40.092 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:49:40.092 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:49:40.093 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:49:40.093 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:49:40.093 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:49:40.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:49:40.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:49:40.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2021-07-26T06:49:40.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:49:40.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:49:40.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:49:40.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:49:40.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:49:40.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:49:41.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:49:41.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:49:41.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:49:41.028 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:49:41.028 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:49:41.028 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:49:41.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:41.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:49:41.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:49:41.052 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: seq 0 9 2021-07-26T06:49:41.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:49:41.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' true = true ']' 2021-07-26T06:49:41.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2021-07-26T06:49:41.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:49:41.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:49:41.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:49:41.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:49:41.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:49:41.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:49:41.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:49:41.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:49:41.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:49:41.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:49:41.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:49:41.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:49:41.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:49:41.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:49:41.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:49:41.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:49:41.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:49:41.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:49:41.062 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:49:41.063 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:49:41.063 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:49:41.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:41.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:41.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:49:41.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:49:41.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:49:41.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:49:41.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:49:41.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:49:41.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:49:41.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:49:41.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:49:41.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:49:41.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:49:41.066 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:49:41.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=28631277-0f8d-45d2-af14-8733f66cb0d5 2021-07-26T06:49:41.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 28631277-0f8d-45d2-af14-8733f66cb0d5' 2021-07-26T06:49:41.067 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 28631277-0f8d-45d2-af14-8733f66cb0d5 2021-07-26T06:49:41.067 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:49:41.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAFW/5gOdVQBBAAShoDrWTX4EiXXJ3oXEZUcg== 2021-07-26T06:49:41.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAFW/5gOdVQBBAAShoDrWTX4EiXXJ3oXEZUcg=="}' 2021-07-26T06:49:41.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 28631277-0f8d-45d2-af14-8733f66cb0d5 -i td/osd-scrub-repair/0/new.json 2021-07-26T06:49:41.374 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:49:41.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:49:41.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAFW/5gOdVQBBAAShoDrWTX4EiXXJ3oXEZUcg== --osd-uuid 28631277-0f8d-45d2-af14-8733f66cb0d5 2021-07-26T06:49:41.410 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:41.410+0000 7f97e61440c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:41.411 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:41.412+0000 7f97e61440c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:41.412 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:41.412+0000 7f97e61440c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:41.413 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:41.413+0000 7f97e61440c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:49:41.413 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:41.413+0000 7f97e61440c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:49:41.413 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:41.413+0000 7f97e61440c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:49:41.491 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:41.491+0000 7f97e61440c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T06:49:47.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:49:47.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:49:47.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:49:47.129 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:49:47.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:49:47.468 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:49:47.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:49:47.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:49:47.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:49:47.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:49:47.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:49:47.490 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:47.491+0000 7f8f0c3480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:47.492 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:47.493+0000 7f8f0c3480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:47.493 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:47.494+0000 7f8f0c3480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:47.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:49:47.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:49:47.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:49:47.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:49:47.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:49:47.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:47.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:49:47.742 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:49:47.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:47.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:49:48.008 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:48.009+0000 7f8f0c3480c0 -1 Falling back to public interface 2021-07-26T06:49:48.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:49:49.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:49:49.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:49.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:49:49.046 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:49:49.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:49.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:49:49.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:49:50.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:49:50.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:50.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:49:50.324 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:49:50.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:50.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:49:50.440 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:50.441+0000 7f8f0c3480c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:49:50.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:49:51.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:49:51.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:51.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:49:51.616 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:49:51.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:51.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:49:51.892 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/257140,v1:127.0.0.1:6803/257140] [v2:127.0.0.1:6804/257140,v1:127.0.0.1:6805/257140] exists,up 28631277-0f8d-45d2-af14-8733f66cb0d5 2021-07-26T06:49:51.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:49:51.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:49:51.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:49:51.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:49:51.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' true = true ']' 2021-07-26T06:49:51.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2021-07-26T06:49:51.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:49:51.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:49:51.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:49:51.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:49:51.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:49:51.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:49:51.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:49:51.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:49:51.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:49:51.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:49:51.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:49:51.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:49:51.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:49:51.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:49:51.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:49:51.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:49:51.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:49:51.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:49:51.902 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:49:51.903 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:49:51.903 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:49:51.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:51.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:49:51.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:49:51.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:49:51.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:49:51.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:49:51.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:49:51.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:49:51.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:49:51.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:49:51.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:49:51.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:49:51.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:49:51.906 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:49:51.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3598d3d1-df6c-4735-bb6d-97d937dd2bcd 2021-07-26T06:49:51.907 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 3598d3d1-df6c-4735-bb6d-97d937dd2bcd 2021-07-26T06:49:51.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3598d3d1-df6c-4735-bb6d-97d937dd2bcd' 2021-07-26T06:49:51.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:49:51.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPW/5g3a5RNhAA46iM55qP09BsXv5xe4PVVQ== 2021-07-26T06:49:51.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPW/5g3a5RNhAA46iM55qP09BsXv5xe4PVVQ=="}' 2021-07-26T06:49:51.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3598d3d1-df6c-4735-bb6d-97d937dd2bcd -i td/osd-scrub-repair/1/new.json 2021-07-26T06:49:52.267 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:49:52.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:49:52.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAPW/5g3a5RNhAA46iM55qP09BsXv5xe4PVVQ== --osd-uuid 3598d3d1-df6c-4735-bb6d-97d937dd2bcd 2021-07-26T06:49:52.304 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:52.305+0000 7f60c9b670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:52.305 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:52.306+0000 7f60c9b670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:52.306 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:52.307+0000 7f60c9b670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:52.307 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:52.308+0000 7f60c9b670c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:49:52.307 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:52.308+0000 7f60c9b670c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:49:52.307 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:52.308+0000 7f60c9b670c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:49:52.375 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:52.376+0000 7f60c9b670c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T06:49:57.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:49:57.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:49:57.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:49:57.823 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:49:57.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:49:58.162 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:49:58.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:49:58.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:49:58.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:49:58.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:49:58.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:49:58.183 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:58.185+0000 7fa0f8f680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:58.206 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:58.207+0000 7fa0f8f680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:58.207 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:58.208+0000 7fa0f8f680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:49:58.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:49:58.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:49:58.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:49:58.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:49:58.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:49:58.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:58.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:49:58.433 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:49:58.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:58.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:49:58.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:49:58.732 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:49:58.734+0000 7fa0f8f680c0 -1 Falling back to public interface 2021-07-26T06:49:59.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:49:59.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:59.709 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:49:59.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:49:59.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:59.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:49:59.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:00.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:00.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:00.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:50:00.987 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:50:00.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:00.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:50:01.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:01.306 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:01.307+0000 7fa0f8f680c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:50:02.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:02.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:02.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:50:02.269 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:50:02.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:02.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:50:02.545 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/258539,v1:127.0.0.1:6811/258539] [v2:127.0.0.1:6812/258539,v1:127.0.0.1:6813/258539] exists,up 3598d3d1-df6c-4735-bb6d-97d937dd2bcd 2021-07-26T06:50:02.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:50:02.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:50:02.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:50:02.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:50:02.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' true = true ']' 2021-07-26T06:50:02.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2021-07-26T06:50:02.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:50:02.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:50:02.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:50:02.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:50:02.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:50:02.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:50:02.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:50:02.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:50:02.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:50:02.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:50:02.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:50:02.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:50:02.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:50:02.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:50:02.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:50:02.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:50:02.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:50:02.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:50:02.556 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:50:02.556 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:50:02.557 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:50:02.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:50:02.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:50:02.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:50:02.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:50:02.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:50:02.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:50:02.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:50:02.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:50:02.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:50:02.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:50:02.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:50:02.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:50:02.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:50:02.560 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:50:02.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1c88a828-5d99-44d2-803f-568f237379c5 2021-07-26T06:50:02.561 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 1c88a828-5d99-44d2-803f-568f237379c5 2021-07-26T06:50:02.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1c88a828-5d99-44d2-803f-568f237379c5' 2021-07-26T06:50:02.561 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:50:02.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaW/5g4FOyIRAAAYAxObaTksgUdzKEkv6+Lg== 2021-07-26T06:50:02.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaW/5g4FOyIRAAAYAxObaTksgUdzKEkv6+Lg=="}' 2021-07-26T06:50:02.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1c88a828-5d99-44d2-803f-568f237379c5 -i td/osd-scrub-repair/2/new.json 2021-07-26T06:50:02.952 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:50:02.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2021-07-26T06:50:02.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAaW/5g4FOyIRAAAYAxObaTksgUdzKEkv6+Lg== --osd-uuid 1c88a828-5d99-44d2-803f-568f237379c5 2021-07-26T06:50:02.987 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:02.988+0000 7f9023a230c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:03.001 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:03.002+0000 7f9023a230c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:03.002 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:03.003+0000 7f9023a230c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:03.002 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:03.003+0000 7f9023a230c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:50:03.003 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:03.003+0000 7f9023a230c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:50:03.003 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:03.003+0000 7f9023a230c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:50:03.080 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:03.081+0000 7f9023a230c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2021-07-26T06:50:08.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T06:50:08.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:50:08.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:50:08.840 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:50:08.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:50:09.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:50:09.178 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:50:09.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:50:09.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:50:09.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:50:09.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:50:09.200 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:09.201+0000 7f4e791dd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:09.201 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:09.203+0000 7f4e791dd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:09.202 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:09.203+0000 7f4e791dd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:09.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:50:09.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:50:09.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:50:09.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:50:09.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:50:09.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:09.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:50:09.491 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:50:09.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:09.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:50:09.718 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:09.719+0000 7f4e791dd0c0 -1 Falling back to public interface 2021-07-26T06:50:09.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:10.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:10.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:10.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:50:10.760 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:50:10.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:10.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:50:11.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:12.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:12.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:12.036 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:50:12.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:50:12.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:12.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:50:12.137 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:12.138+0000 7f4e791dd0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:50:12.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:13.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:13.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:13.318 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:50:13.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:50:13.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:13.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:50:13.620 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/259942,v1:127.0.0.1:6819/259942] [v2:127.0.0.1:6820/259942,v1:127.0.0.1:6821/259942] exists,up 1c88a828-5d99-44d2-803f-568f237379c5 2021-07-26T06:50:13.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:50:13.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:50:13.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:50:13.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:50:13.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' true = true ']' 2021-07-26T06:50:13.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2021-07-26T06:50:13.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:50:13.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:50:13.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T06:50:13.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:50:13.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:50:13.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:50:13.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:50:13.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:50:13.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:50:13.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:50:13.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:50:13.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:50:13.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:50:13.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:50:13.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:50:13.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:50:13.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:50:13.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:50:13.651 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:50:13.651 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:50:13.651 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:50:13.652 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:50:13.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:50:13.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:50:13.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:50:13.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:50:13.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:50:13.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:50:13.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:50:13.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:50:13.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:50:13.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:50:13.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:50:13.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:50:13.655 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:50:13.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c55da62-886d-4a69-a717-2a90c8e238a9 2021-07-26T06:50:13.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9c55da62-886d-4a69-a717-2a90c8e238a9' 2021-07-26T06:50:13.658 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 9c55da62-886d-4a69-a717-2a90c8e238a9 2021-07-26T06:50:13.659 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:50:13.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlW/5gtkMfJhAAmx8xJOJSL/YhMTT+YmcMdw== 2021-07-26T06:50:13.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlW/5gtkMfJhAAmx8xJOJSL/YhMTT+YmcMdw=="}' 2021-07-26T06:50:13.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c55da62-886d-4a69-a717-2a90c8e238a9 -i td/osd-scrub-repair/3/new.json 2021-07-26T06:50:14.071 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:50:14.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2021-07-26T06:50:14.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAlW/5gtkMfJhAAmx8xJOJSL/YhMTT+YmcMdw== --osd-uuid 9c55da62-886d-4a69-a717-2a90c8e238a9 2021-07-26T06:50:14.107 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:14.108+0000 7f3d5e8d00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:14.129 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:14.131+0000 7f3d5e8d00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:14.130 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:14.131+0000 7f3d5e8d00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:14.131 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:14.132+0000 7f3d5e8d00c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2021-07-26T06:50:14.131 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:14.132+0000 7f3d5e8d00c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2021-07-26T06:50:14.131 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:14.132+0000 7f3d5e8d00c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2021-07-26T06:50:14.209 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:14.210+0000 7f3d5e8d00c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2021-07-26T06:50:19.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2021-07-26T06:50:19.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:50:19.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T06:50:19.922 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd3 key to auth repository 2021-07-26T06:50:19.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:50:20.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T06:50:20.272 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.3 2021-07-26T06:50:20.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:50:20.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:50:20.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:50:20.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:50:20.294 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:20.295+0000 7f868334c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:20.319 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:20.320+0000 7f868334c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:20.319 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:20.320+0000 7f868334c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:20.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T06:50:20.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:50:20.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:50:20.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:50:20.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:50:20.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:20.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:50:20.563 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:50:20.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:20.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:50:20.836 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:20.836+0000 7f868334c0c0 -1 Falling back to public interface 2021-07-26T06:50:20.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:21.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:21.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:21.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:50:21.865 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:50:21.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:21.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:50:22.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:23.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:23.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:23.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:50:23.160 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:50:23.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:23.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:50:23.197 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:23.197+0000 7f868334c0c0 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T06:50:23.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:24.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:24.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:24.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:50:24.515 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:50:24.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:24.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:50:24.815 INFO:tasks.workunit.client.0.smithi071.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/261349,v1:127.0.0.1:6827/261349] [v2:127.0.0.1:6828/261349,v1:127.0.0.1:6829/261349] exists,up 9c55da62-886d-4a69-a717-2a90c8e238a9 2021-07-26T06:50:24.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:50:24.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:50:24.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:50:24.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:50:24.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' true = true ']' 2021-07-26T06:50:24.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2021-07-26T06:50:24.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:50:24.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:50:24.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-07-26T06:50:24.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:50:24.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2021-07-26T06:50:24.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:50:24.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:50:24.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:50:24.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:50:24.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2021-07-26T06:50:24.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2021-07-26T06:50:24.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:50:24.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:50:24.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:50:24.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:50:24.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:50:24.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:50:24.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:50:24.828 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:50:24.829 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:50:24.829 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:50:24.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:50:24.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:50:24.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:50:24.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:50:24.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:50:24.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:50:24.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:50:24.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:50:24.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:50:24.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:50:24.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:50:24.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:50:24.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2021-07-26T06:50:24.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:50:24.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f559e811-75d6-4cf2-bb0c-03342f47b6bc 2021-07-26T06:50:24.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 f559e811-75d6-4cf2-bb0c-03342f47b6bc' 2021-07-26T06:50:24.833 INFO:tasks.workunit.client.0.smithi071.stdout:add osd4 f559e811-75d6-4cf2-bb0c-03342f47b6bc 2021-07-26T06:50:24.834 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:50:24.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwW/5gAOHGMRAA1cL15ofIf3vF8VpSF6sEFw== 2021-07-26T06:50:24.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwW/5gAOHGMRAA1cL15ofIf3vF8VpSF6sEFw=="}' 2021-07-26T06:50:24.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f559e811-75d6-4cf2-bb0c-03342f47b6bc -i td/osd-scrub-repair/4/new.json 2021-07-26T06:50:25.225 INFO:tasks.workunit.client.0.smithi071.stdout:4 2021-07-26T06:50:25.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2021-07-26T06:50:25.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAwW/5gAOHGMRAA1cL15ofIf3vF8VpSF6sEFw== --osd-uuid f559e811-75d6-4cf2-bb0c-03342f47b6bc 2021-07-26T06:50:25.261 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:25.261+0000 7fe56f3270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:25.284 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:25.284+0000 7fe56f3270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:25.284 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:25.285+0000 7fe56f3270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:25.285 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:25.285+0000 7fe56f3270c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2021-07-26T06:50:25.286 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:25.285+0000 7fe56f3270c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2021-07-26T06:50:25.286 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:25.285+0000 7fe56f3270c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2021-07-26T06:50:25.362 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:25.362+0000 7fe56f3270c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2021-07-26T06:50:31.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2021-07-26T06:50:31.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:50:31.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2021-07-26T06:50:31.444 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd4 key to auth repository 2021-07-26T06:50:31.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:50:31.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2021-07-26T06:50:31.785 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.4 2021-07-26T06:50:31.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:50:31.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:50:31.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:50:31.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:50:31.806 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:31.806+0000 7fda514e00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:31.808 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:31.808+0000 7fda514e00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:31.809 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:31.809+0000 7fda514e00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:32.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2021-07-26T06:50:32.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:50:32.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=4 2021-07-26T06:50:32.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:50:32.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:50:32.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:32.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:50:32.207 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:50:32.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:32.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:50:32.330 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:32.330+0000 7fda514e00c0 -1 Falling back to public interface 2021-07-26T06:50:32.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:33.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:33.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:33.476 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:50:33.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:50:33.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:33.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:50:33.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:34.677 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:34.677+0000 7fda514e00c0 -1 osd.4 0 log_to_monitors {default=true} 2021-07-26T06:50:34.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:34.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:34.754 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:50:34.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:50:34.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:34.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:50:35.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:36.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:36.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:36.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:50:36.087 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:50:36.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:36.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:50:36.382 INFO:tasks.workunit.client.0.smithi071.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/262748,v1:127.0.0.1:6835/262748] [v2:127.0.0.1:6836/262748,v1:127.0.0.1:6837/262748] exists,up f559e811-75d6-4cf2-bb0c-03342f47b6bc 2021-07-26T06:50:36.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:50:36.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:50:36.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:50:36.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:50:36.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' true = true ']' 2021-07-26T06:50:36.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2021-07-26T06:50:36.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:50:36.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:50:36.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2021-07-26T06:50:36.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:50:36.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:50:36.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:50:36.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:50:36.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:50:36.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:50:36.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2021-07-26T06:50:36.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2021-07-26T06:50:36.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:50:36.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:50:36.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:50:36.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:50:36.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:50:36.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:50:36.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:50:36.392 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:50:36.392 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:50:36.393 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:50:36.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:50:36.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:50:36.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:50:36.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:50:36.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:50:36.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:50:36.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:50:36.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:50:36.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:50:36.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:50:36.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:50:36.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:50:36.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2021-07-26T06:50:36.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:50:36.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bbee9dc4-1bba-496c-96c5-7f58e2aee6bc 2021-07-26T06:50:36.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 bbee9dc4-1bba-496c-96c5-7f58e2aee6bc' 2021-07-26T06:50:36.397 INFO:tasks.workunit.client.0.smithi071.stdout:add osd5 bbee9dc4-1bba-496c-96c5-7f58e2aee6bc 2021-07-26T06:50:36.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:50:36.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8W/5gWk3qFxAACc8yPqWECl/b6Vz6X1oYWw== 2021-07-26T06:50:36.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8W/5gWk3qFxAACc8yPqWECl/b6Vz6X1oYWw=="}' 2021-07-26T06:50:36.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bbee9dc4-1bba-496c-96c5-7f58e2aee6bc -i td/osd-scrub-repair/5/new.json 2021-07-26T06:50:36.789 INFO:tasks.workunit.client.0.smithi071.stdout:5 2021-07-26T06:50:36.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2021-07-26T06:50:36.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA8W/5gWk3qFxAACc8yPqWECl/b6Vz6X1oYWw== --osd-uuid bbee9dc4-1bba-496c-96c5-7f58e2aee6bc 2021-07-26T06:50:36.825 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:36.825+0000 7fdab17830c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:36.847 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:36.847+0000 7fdab17830c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:36.849 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:36.848+0000 7fdab17830c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:36.849 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:36.849+0000 7fdab17830c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2021-07-26T06:50:36.849 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:36.849+0000 7fdab17830c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2021-07-26T06:50:36.849 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:36.849+0000 7fdab17830c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2021-07-26T06:50:36.934 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:36.935+0000 7fdab17830c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2021-07-26T06:50:42.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2021-07-26T06:50:42.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:50:42.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2021-07-26T06:50:42.641 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd5 key to auth repository 2021-07-26T06:50:42.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:50:42.979 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.5 2021-07-26T06:50:42.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2021-07-26T06:50:42.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:50:42.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:50:42.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:50:42.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:50:43.001 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:43.002+0000 7f744f4050c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:43.003 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:43.004+0000 7f744f4050c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:43.004 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:43.005+0000 7f744f4050c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:43.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2021-07-26T06:50:43.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:50:43.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:50:43.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:50:43.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:50:43.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:43.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:50:43.254 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:50:43.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:43.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:50:43.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:43.539 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:43.541+0000 7f744f4050c0 -1 Falling back to public interface 2021-07-26T06:50:44.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:44.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:44.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:50:44.521 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:50:44.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:44.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:50:44.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:45.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:45.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:45.797 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:50:45.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:50:45.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:45.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:50:45.999 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:46.000+0000 7f744f4050c0 -1 osd.5 0 log_to_monitors {default=true} 2021-07-26T06:50:46.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:47.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:47.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:47.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:50:47.208 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:50:47.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:47.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:50:47.485 INFO:tasks.workunit.client.0.smithi071.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/264147,v1:127.0.0.1:6843/264147] [v2:127.0.0.1:6844/264147,v1:127.0.0.1:6845/264147] exists,up bbee9dc4-1bba-496c-96c5-7f58e2aee6bc 2021-07-26T06:50:47.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:50:47.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:50:47.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:50:47.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:50:47.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' true = true ']' 2021-07-26T06:50:47.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2021-07-26T06:50:47.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:50:47.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:50:47.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2021-07-26T06:50:47.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:50:47.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2021-07-26T06:50:47.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:50:47.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:50:47.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:50:47.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:50:47.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2021-07-26T06:50:47.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2021-07-26T06:50:47.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:50:47.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:50:47.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:50:47.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:50:47.496 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:50:47.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:50:47.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:50:47.497 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:50:47.497 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:50:47.498 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:50:47.498 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:50:47.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:50:47.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:50:47.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:50:47.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:50:47.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:50:47.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:50:47.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:50:47.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:50:47.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:50:47.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:50:47.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:50:47.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2021-07-26T06:50:47.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:50:47.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fac931bf-c434-4f33-a1c3-166950821e45 2021-07-26T06:50:47.502 INFO:tasks.workunit.client.0.smithi071.stdout:add osd6 fac931bf-c434-4f33-a1c3-166950821e45 2021-07-26T06:50:47.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 fac931bf-c434-4f33-a1c3-166950821e45' 2021-07-26T06:50:47.503 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:50:47.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHW/5gNsIUHhAAuw8bdWgOwQ04JPT6hM/WVg== 2021-07-26T06:50:47.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHW/5gNsIUHhAAuw8bdWgOwQ04JPT6hM/WVg=="}' 2021-07-26T06:50:47.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fac931bf-c434-4f33-a1c3-166950821e45 -i td/osd-scrub-repair/6/new.json 2021-07-26T06:50:47.903 INFO:tasks.workunit.client.0.smithi071.stdout:6 2021-07-26T06:50:47.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2021-07-26T06:50:47.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBHW/5gNsIUHhAAuw8bdWgOwQ04JPT6hM/WVg== --osd-uuid fac931bf-c434-4f33-a1c3-166950821e45 2021-07-26T06:50:47.940 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:47.941+0000 7f45b37490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:47.961 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:47.962+0000 7f45b37490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:47.962 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:47.963+0000 7f45b37490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:47.962 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:47.963+0000 7f45b37490c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2021-07-26T06:50:47.963 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:47.963+0000 7f45b37490c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2021-07-26T06:50:47.963 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:47.963+0000 7f45b37490c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2021-07-26T06:50:48.037 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:48.038+0000 7f45b37490c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2021-07-26T06:50:53.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2021-07-26T06:50:53.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:50:53.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2021-07-26T06:50:53.783 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd6 key to auth repository 2021-07-26T06:50:53.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:50:54.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2021-07-26T06:50:54.127 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.6 2021-07-26T06:50:54.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:50:54.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:50:54.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:50:54.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:50:54.148 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:54.149+0000 7f4e191c20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:54.151 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:54.152+0000 7f4e191c20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:54.152 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:54.153+0000 7f4e191c20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:54.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2021-07-26T06:50:54.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:50:54.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=6 2021-07-26T06:50:54.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:50:54.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:50:54.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:54.396 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:50:54.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:50:54.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:54.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.6 up' 2021-07-26T06:50:54.667 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:54.669+0000 7f4e191c20c0 -1 Falling back to public interface 2021-07-26T06:50:54.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:55.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:55.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:55.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:50:55.707 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:50:55.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:55.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.6 up' 2021-07-26T06:50:55.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:56.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:56.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:56.988 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:50:56.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:50:56.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:56.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.6 up' 2021-07-26T06:50:57.031 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:57.032+0000 7f4e191c20c0 -1 osd.6 0 log_to_monitors {default=true} 2021-07-26T06:50:57.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:50:58.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:50:58.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:50:58.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:50:58.347 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:50:58.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:50:58.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.6 up' 2021-07-26T06:50:58.621 INFO:tasks.workunit.client.0.smithi071.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/265546,v1:127.0.0.1:6851/265546] [v2:127.0.0.1:6852/265546,v1:127.0.0.1:6853/265546] exists,up fac931bf-c434-4f33-a1c3-166950821e45 2021-07-26T06:50:58.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:50:58.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:50:58.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:50:58.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:50:58.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' true = true ']' 2021-07-26T06:50:58.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2021-07-26T06:50:58.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:50:58.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:50:58.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2021-07-26T06:50:58.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:50:58.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2021-07-26T06:50:58.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:50:58.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:50:58.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:50:58.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:50:58.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2021-07-26T06:50:58.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2021-07-26T06:50:58.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:50:58.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:50:58.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:50:58.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:50:58.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:50:58.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:50:58.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:50:58.635 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:50:58.636 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:50:58.636 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:50:58.637 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:50:58.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:50:58.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:50:58.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:50:58.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:50:58.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:50:58.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:50:58.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:50:58.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:50:58.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:50:58.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:50:58.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:50:58.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2021-07-26T06:50:58.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:50:58.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=44b92b47-fcbc-4c79-8441-fc23f2a615c9 2021-07-26T06:50:58.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 44b92b47-fcbc-4c79-8441-fc23f2a615c9' 2021-07-26T06:50:58.641 INFO:tasks.workunit.client.0.smithi071.stdout:add osd7 44b92b47-fcbc-4c79-8441-fc23f2a615c9 2021-07-26T06:50:58.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:50:58.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSW/5g/gc1JhAAAHOwdgoInWUq2thFletYRQ== 2021-07-26T06:50:58.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSW/5g/gc1JhAAAHOwdgoInWUq2thFletYRQ=="}' 2021-07-26T06:50:58.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 44b92b47-fcbc-4c79-8441-fc23f2a615c9 -i td/osd-scrub-repair/7/new.json 2021-07-26T06:50:59.062 INFO:tasks.workunit.client.0.smithi071.stdout:7 2021-07-26T06:50:59.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2021-07-26T06:50:59.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBSW/5g/gc1JhAAAHOwdgoInWUq2thFletYRQ== --osd-uuid 44b92b47-fcbc-4c79-8441-fc23f2a615c9 2021-07-26T06:50:59.099 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:59.099+0000 7f4f9be200c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:59.111 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:59.113+0000 7f4f9be200c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:59.113 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:59.114+0000 7f4f9be200c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:50:59.114 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:59.114+0000 7f4f9be200c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2021-07-26T06:50:59.114 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:59.114+0000 7f4f9be200c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2021-07-26T06:50:59.114 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:59.114+0000 7f4f9be200c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2021-07-26T06:50:59.187 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:50:59.188+0000 7f4f9be200c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2021-07-26T06:51:05.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2021-07-26T06:51:05.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:51:05.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2021-07-26T06:51:05.011 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd7 key to auth repository 2021-07-26T06:51:05.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:51:05.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2021-07-26T06:51:05.349 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.7 2021-07-26T06:51:05.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:51:05.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:51:05.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:51:05.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:51:05.370 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:05.372+0000 7f87718430c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:05.372 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:05.373+0000 7f87718430c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:05.373 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:05.374+0000 7f87718430c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:05.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2021-07-26T06:51:05.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:51:05.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=7 2021-07-26T06:51:05.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:51:05.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:51:05.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:05.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:51:05.622 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:51:05.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:05.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.7 up' 2021-07-26T06:51:05.892 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:05.893+0000 7f87718430c0 -1 Falling back to public interface 2021-07-26T06:51:05.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:51:06.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:51:06.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:06.903 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:51:06.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:51:06.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:06.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.7 up' 2021-07-26T06:51:07.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:51:08.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:51:08.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:08.185 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:51:08.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:51:08.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:08.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.7 up' 2021-07-26T06:51:08.377 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:08.378+0000 7f87718430c0 -1 osd.7 0 log_to_monitors {default=true} 2021-07-26T06:51:08.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:51:09.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:51:09.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:09.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:51:09.502 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:51:09.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:09.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.7 up' 2021-07-26T06:51:09.779 INFO:tasks.workunit.client.0.smithi071.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/266946,v1:127.0.0.1:6859/266946] [v2:127.0.0.1:6860/266946,v1:127.0.0.1:6861/266946] exists,up 44b92b47-fcbc-4c79-8441-fc23f2a615c9 2021-07-26T06:51:09.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:51:09.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:51:09.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:51:09.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:51:09.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' true = true ']' 2021-07-26T06:51:09.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2021-07-26T06:51:09.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:51:09.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:51:09.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2021-07-26T06:51:09.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:51:09.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2021-07-26T06:51:09.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:51:09.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:51:09.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:51:09.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:51:09.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2021-07-26T06:51:09.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2021-07-26T06:51:09.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:51:09.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:51:09.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:51:09.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:51:09.791 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:51:09.791 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:51:09.791 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:51:09.791 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:51:09.792 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:51:09.792 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:51:09.792 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:51:09.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:51:09.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:51:09.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:51:09.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:51:09.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:51:09.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:51:09.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:51:09.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:51:09.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:51:09.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:51:09.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:51:09.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2021-07-26T06:51:09.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:51:09.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7c6767e6-d65a-4779-b22e-bda16d2c93b4 2021-07-26T06:51:09.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 7c6767e6-d65a-4779-b22e-bda16d2c93b4' 2021-07-26T06:51:09.796 INFO:tasks.workunit.client.0.smithi071.stdout:add osd8 7c6767e6-d65a-4779-b22e-bda16d2c93b4 2021-07-26T06:51:09.797 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:51:09.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBdW/5gxp2fLxAA2ywtF6OVOwdHswOmaUkyXg== 2021-07-26T06:51:09.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBdW/5gxp2fLxAA2ywtF6OVOwdHswOmaUkyXg=="}' 2021-07-26T06:51:09.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7c6767e6-d65a-4779-b22e-bda16d2c93b4 -i td/osd-scrub-repair/8/new.json 2021-07-26T06:51:10.148 INFO:tasks.workunit.client.0.smithi071.stdout:8 2021-07-26T06:51:10.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2021-07-26T06:51:10.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBdW/5gxp2fLxAA2ywtF6OVOwdHswOmaUkyXg== --osd-uuid 7c6767e6-d65a-4779-b22e-bda16d2c93b4 2021-07-26T06:51:10.188 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:10.186+0000 7fc4a29350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:10.189 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:10.187+0000 7fc4a29350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:10.189 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:10.188+0000 7fc4a29350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:10.189 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:10.189+0000 7fc4a29350c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2021-07-26T06:51:10.190 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:10.189+0000 7fc4a29350c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2021-07-26T06:51:10.190 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:10.189+0000 7fc4a29350c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2021-07-26T06:51:10.299 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:10.299+0000 7fc4a29350c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2021-07-26T06:51:16.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2021-07-26T06:51:16.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:51:16.004 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd8 key to auth repository 2021-07-26T06:51:16.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2021-07-26T06:51:16.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:51:16.346 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.8 2021-07-26T06:51:16.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2021-07-26T06:51:16.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:51:16.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:51:16.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:51:16.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:51:16.369 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:16.370+0000 7f75b10eb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:16.370 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:16.372+0000 7f75b10eb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:16.371 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:16.372+0000 7f75b10eb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:16.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2021-07-26T06:51:16.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:51:16.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=8 2021-07-26T06:51:16.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:51:16.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:51:16.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:16.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:51:16.618 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:51:16.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:16.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.8 up' 2021-07-26T06:51:16.885 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:16.887+0000 7f75b10eb0c0 -1 Falling back to public interface 2021-07-26T06:51:16.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:51:17.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:51:17.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:17.925 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:51:17.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:51:17.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:17.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.8 up' 2021-07-26T06:51:18.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:51:19.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:51:19.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:19.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:51:19.230 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:51:19.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:19.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.8 up' 2021-07-26T06:51:19.240 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:19.240+0000 7f75b10eb0c0 -1 osd.8 0 log_to_monitors {default=true} 2021-07-26T06:51:19.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:51:20.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:51:20.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:20.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:51:20.558 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:51:20.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:20.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.8 up' 2021-07-26T06:51:20.861 INFO:tasks.workunit.client.0.smithi071.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/268345,v1:127.0.0.1:6867/268345] [v2:127.0.0.1:6868/268345,v1:127.0.0.1:6869/268345] exists,up 7c6767e6-d65a-4779-b22e-bda16d2c93b4 2021-07-26T06:51:20.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:51:20.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:51:20.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:51:20.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: for id in $(seq 0 9) 2021-07-26T06:51:20.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:833: corrupt_and_repair_lrc: '[' true = true ']' 2021-07-26T06:51:20.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2021-07-26T06:51:20.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:51:20.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:51:20.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2021-07-26T06:51:20.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:51:20.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:51:20.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:51:20.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:51:20.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:51:20.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:51:20.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2021-07-26T06:51:20.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2021-07-26T06:51:20.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:51:20.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:51:20.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:51:20.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:51:20.871 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:51:20.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:51:20.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:51:20.872 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:51:20.872 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:51:20.872 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:51:20.873 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:51:20.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:51:20.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:51:20.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:51:20.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:51:20.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:51:20.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:51:20.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:51:20.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:51:20.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:51:20.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:51:20.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:51:20.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2021-07-26T06:51:20.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:51:20.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1b1672b3-c8a7-4987-b315-589891f1b185 2021-07-26T06:51:20.876 INFO:tasks.workunit.client.0.smithi071.stdout:add osd9 1b1672b3-c8a7-4987-b315-589891f1b185 2021-07-26T06:51:20.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 1b1672b3-c8a7-4987-b315-589891f1b185' 2021-07-26T06:51:20.877 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:51:20.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBoW/5g49Z/NBAAL5DOZ3aogepwGoDVgI19Gw== 2021-07-26T06:51:20.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBoW/5g49Z/NBAAL5DOZ3aogepwGoDVgI19Gw=="}' 2021-07-26T06:51:20.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1b1672b3-c8a7-4987-b315-589891f1b185 -i td/osd-scrub-repair/9/new.json 2021-07-26T06:51:21.264 INFO:tasks.workunit.client.0.smithi071.stdout:9 2021-07-26T06:51:21.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2021-07-26T06:51:21.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBoW/5g49Z/NBAAL5DOZ3aogepwGoDVgI19Gw== --osd-uuid 1b1672b3-c8a7-4987-b315-589891f1b185 2021-07-26T06:51:21.300 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:21.300+0000 7f535bc690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:21.328 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:21.328+0000 7f535bc690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:21.329 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:21.329+0000 7f535bc690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:21.330 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:21.329+0000 7f535bc690c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2021-07-26T06:51:21.330 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:21.329+0000 7f535bc690c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2021-07-26T06:51:21.330 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:21.329+0000 7f535bc690c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2021-07-26T06:51:21.425 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:21.425+0000 7f535bc690c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2021-07-26T06:51:27.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2021-07-26T06:51:27.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:51:27.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2021-07-26T06:51:27.506 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd9 key to auth repository 2021-07-26T06:51:27.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:51:27.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2021-07-26T06:51:27.847 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.9 2021-07-26T06:51:27.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:51:27.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:51:27.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:51:27.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:51:27.869 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:27.869+0000 7f72062810c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:27.904 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:27.904+0000 7f72062810c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:27.904 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:27.905+0000 7f72062810c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:28.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2021-07-26T06:51:28.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:51:28.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=9 2021-07-26T06:51:28.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:51:28.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:51:28.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:28.177 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:51:28.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:51:28.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:28.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:51:28.424 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:28.424+0000 7f72062810c0 -1 Falling back to public interface 2021-07-26T06:51:28.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:51:29.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:51:29.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:29.448 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:51:29.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:51:29.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:29.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:51:29.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:51:30.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:51:30.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:30.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:51:30.737 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:51:30.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:30.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:51:30.810 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:30.810+0000 7f72062810c0 -1 osd.9 0 log_to_monitors {default=true} 2021-07-26T06:51:31.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:51:32.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:51:32.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:32.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:51:32.043 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:51:32.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:32.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:51:32.372 INFO:tasks.workunit.client.0.smithi071.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/269752,v1:127.0.0.1:6875/269752] [v2:127.0.0.1:6876/269752,v1:127.0.0.1:6877/269752] exists,up 1b1672b3-c8a7-4987-b315-589891f1b185 2021-07-26T06:51:32.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:51:32.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:51:32.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:51:32.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: corrupt_and_repair_lrc: create_rbd_pool 2021-07-26T06:51:32.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:51:32.635 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T06:51:32.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:51:32.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:51:33.288 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T06:51:33.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:51:34.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:51:36.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: corrupt_and_repair_lrc: wait_for_clean 2021-07-26T06:51:36.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:51:36.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:51:36.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:51:36.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:51:36.682 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:51:36.682 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:51:36.683 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:51:36.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:51:36.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:51:36.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:51:36.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:51:36.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:51:36.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:51:36.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:51:36.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:51:37.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:51:37.059 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:51:37.059 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:51:37.059 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:51:37.059 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:51:37.060 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:51:37.060 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:51:37.061 INFO:tasks.workunit.client.0.smithi071.stderr:7 2021-07-26T06:51:37.061 INFO:tasks.workunit.client.0.smithi071.stderr:8 2021-07-26T06:51:37.061 INFO:tasks.workunit.client.0.smithi071.stderr:9' 2021-07-26T06:51:37.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:51:37.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:37.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:51:37.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836503 2021-07-26T06:51:37.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836503 2021-07-26T06:51:37.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503' 2021-07-26T06:51:37.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:37.220 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:51:37.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672980 2021-07-26T06:51:37.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672980 2021-07-26T06:51:37.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672980' 2021-07-26T06:51:37.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:37.339 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:51:37.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509459 2021-07-26T06:51:37.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509459 2021-07-26T06:51:37.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509459' 2021-07-26T06:51:37.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:37.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:51:37.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345937 2021-07-26T06:51:37.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345937 2021-07-26T06:51:37.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509459 3-85899345937' 2021-07-26T06:51:37.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:37.576 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:51:37.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182414 2021-07-26T06:51:37.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182414 2021-07-26T06:51:37.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509459 3-85899345937 4-107374182414' 2021-07-26T06:51:37.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:37.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:51:37.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018892 2021-07-26T06:51:37.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018892 2021-07-26T06:51:37.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509459 3-85899345937 4-107374182414 5-128849018892' 2021-07-26T06:51:37.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:37.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:51:37.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855369 2021-07-26T06:51:37.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855369 2021-07-26T06:51:37.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509459 3-85899345937 4-107374182414 5-128849018892 6-150323855369' 2021-07-26T06:51:37.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:37.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:51:38.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691848 2021-07-26T06:51:38.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691848 2021-07-26T06:51:38.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509459 3-85899345937 4-107374182414 5-128849018892 6-150323855369 7-171798691848' 2021-07-26T06:51:38.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:38.049 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:51:38.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528325 2021-07-26T06:51:38.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528325 2021-07-26T06:51:38.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509459 3-85899345937 4-107374182414 5-128849018892 6-150323855369 7-171798691848 8-193273528325' 2021-07-26T06:51:38.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:38.178 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:51:38.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=214748364804 2021-07-26T06:51:38.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 214748364804 2021-07-26T06:51:38.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509459 3-85899345937 4-107374182414 5-128849018892 6-150323855369 7-171798691848 8-193273528325 9-214748364804' 2021-07-26T06:51:38.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:38.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836503 2021-07-26T06:51:38.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:38.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:51:38.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836503 2021-07-26T06:51:38.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:38.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836503 2021-07-26T06:51:38.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2021-07-26T06:51:38.303 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836503 2021-07-26T06:51:38.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:51:38.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836503 -lt 21474836503 2021-07-26T06:51:38.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:38.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672980 2021-07-26T06:51:38.569 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:38.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:51:38.570 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672980 2021-07-26T06:51:38.570 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:38.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672980 2021-07-26T06:51:38.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2021-07-26T06:51:38.572 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672980 2021-07-26T06:51:38.572 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:51:38.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672980 -lt 42949672980 2021-07-26T06:51:38.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:38.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509459 2021-07-26T06:51:38.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:38.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:51:38.905 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509459 2021-07-26T06:51:38.905 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:38.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509459 2021-07-26T06:51:38.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2021-07-26T06:51:38.906 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509459 2021-07-26T06:51:38.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:51:39.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509459 -lt 64424509459 2021-07-26T06:51:39.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:39.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345937 2021-07-26T06:51:39.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:39.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:51:39.174 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345937 2021-07-26T06:51:39.175 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:39.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345937 2021-07-26T06:51:39.176 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.3 seq 85899345937 2021-07-26T06:51:39.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2021-07-26T06:51:39.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:51:39.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345937 -lt 85899345937 2021-07-26T06:51:39.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:39.482 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182414 2021-07-26T06:51:39.482 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:39.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:51:39.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182414 2021-07-26T06:51:39.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:39.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182414 2021-07-26T06:51:39.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2021-07-26T06:51:39.485 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.4 seq 107374182414 2021-07-26T06:51:39.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:51:39.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182414 -lt 107374182414 2021-07-26T06:51:39.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:39.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018892 2021-07-26T06:51:39.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:39.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:51:39.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018892 2021-07-26T06:51:39.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:39.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018892 2021-07-26T06:51:39.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018892' 2021-07-26T06:51:39.757 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.5 seq 128849018892 2021-07-26T06:51:39.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:51:40.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018892 -lt 128849018892 2021-07-26T06:51:40.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:40.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855369 2021-07-26T06:51:40.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:40.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:51:40.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855369 2021-07-26T06:51:40.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:40.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855369 2021-07-26T06:51:40.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855369' 2021-07-26T06:51:40.042 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.6 seq 150323855369 2021-07-26T06:51:40.042 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:51:40.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855370 -lt 150323855369 2021-07-26T06:51:40.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:40.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691848 2021-07-26T06:51:40.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:40.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:51:40.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691848 2021-07-26T06:51:40.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:40.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691848 2021-07-26T06:51:40.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691848' 2021-07-26T06:51:40.314 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.7 seq 171798691848 2021-07-26T06:51:40.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:51:40.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691848 -lt 171798691848 2021-07-26T06:51:40.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:40.583 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528325 2021-07-26T06:51:40.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:40.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:51:40.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528325 2021-07-26T06:51:40.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:40.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528325 2021-07-26T06:51:40.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528325' 2021-07-26T06:51:40.585 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.8 seq 193273528325 2021-07-26T06:51:40.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:51:40.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528325 -lt 193273528325 2021-07-26T06:51:40.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:40.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-214748364804 2021-07-26T06:51:40.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:40.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:51:40.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-214748364804 2021-07-26T06:51:40.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:40.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=214748364804 2021-07-26T06:51:40.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 214748364804' 2021-07-26T06:51:40.914 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.9 seq 214748364804 2021-07-26T06:51:40.915 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:51:41.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 214748364804 -lt 214748364804 2021-07-26T06:51:41.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:51:41.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:51:41.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:51:41.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:51:41.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:51:41.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:51:41.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:51:41.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:51:41.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:51:41.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:51:41.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:51:41.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:51:41.811 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:51:41.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:51:41.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:51:42.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:51:42.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:51:42.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:51:42.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2021-07-26T06:51:42.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=ecpool 2021-07-26T06:51:42.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T06:51:42.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=true 2021-07-26T06:51:42.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T06:51:42.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2021-07-26T06:51:42.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2021-07-26T06:51:42.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2021-07-26T06:51:43.423 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' created 2021-07-26T06:51:43.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:51:44.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' true = true ']' 2021-07-26T06:51:44.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2021-07-26T06:51:44.953 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 2 allow_ec_overwrites to true 2021-07-26T06:51:44.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T06:51:44.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:51:44.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:51:44.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:51:44.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:51:44.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:51:44.965 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:51:44.966 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:51:44.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:51:44.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:51:44.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:51:45.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:51:45.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:51:45.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:51:45.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:51:45.060 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:51:45.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:51:45.488 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:51:45.488 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:51:45.489 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:51:45.489 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:51:45.489 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:51:45.489 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:51:45.489 INFO:tasks.workunit.client.0.smithi071.stderr:7 2021-07-26T06:51:45.490 INFO:tasks.workunit.client.0.smithi071.stderr:8 2021-07-26T06:51:45.490 INFO:tasks.workunit.client.0.smithi071.stderr:9' 2021-07-26T06:51:45.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:51:45.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:45.491 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:51:45.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836505 2021-07-26T06:51:45.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836505 2021-07-26T06:51:45.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505' 2021-07-26T06:51:45.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:45.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:51:45.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672983 2021-07-26T06:51:45.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672983 2021-07-26T06:51:45.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672983' 2021-07-26T06:51:45.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:45.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:51:45.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509462 2021-07-26T06:51:45.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509462 2021-07-26T06:51:45.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509462' 2021-07-26T06:51:45.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:45.888 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:51:46.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345940 2021-07-26T06:51:46.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345940 2021-07-26T06:51:46.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509462 3-85899345940' 2021-07-26T06:51:46.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:46.008 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:51:46.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182417 2021-07-26T06:51:46.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182417 2021-07-26T06:51:46.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509462 3-85899345940 4-107374182417' 2021-07-26T06:51:46.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:46.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:51:46.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018894 2021-07-26T06:51:46.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018894 2021-07-26T06:51:46.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509462 3-85899345940 4-107374182417 5-128849018894' 2021-07-26T06:51:46.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:46.255 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:51:46.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855372 2021-07-26T06:51:46.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855372 2021-07-26T06:51:46.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509462 3-85899345940 4-107374182417 5-128849018894 6-150323855372' 2021-07-26T06:51:46.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:46.374 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:51:46.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691851 2021-07-26T06:51:46.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691851 2021-07-26T06:51:46.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509462 3-85899345940 4-107374182417 5-128849018894 6-150323855372 7-171798691851' 2021-07-26T06:51:46.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:46.492 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:51:46.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528328 2021-07-26T06:51:46.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528328 2021-07-26T06:51:46.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509462 3-85899345940 4-107374182417 5-128849018894 6-150323855372 7-171798691851 8-193273528328' 2021-07-26T06:51:46.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:51:46.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:51:46.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=214748364806 2021-07-26T06:51:46.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 214748364806 2021-07-26T06:51:46.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509462 3-85899345940 4-107374182417 5-128849018894 6-150323855372 7-171798691851 8-193273528328 9-214748364806' 2021-07-26T06:51:46.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:46.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836505 2021-07-26T06:51:46.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:46.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:51:46.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836505 2021-07-26T06:51:46.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:46.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836505 2021-07-26T06:51:46.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2021-07-26T06:51:46.732 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836505 2021-07-26T06:51:46.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:51:47.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836505 -lt 21474836505 2021-07-26T06:51:47.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:47.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672983 2021-07-26T06:51:47.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:47.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:51:47.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672983 2021-07-26T06:51:47.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:47.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672983 2021-07-26T06:51:47.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2021-07-26T06:51:47.004 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672983 2021-07-26T06:51:47.005 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:51:47.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672982 -lt 42949672983 2021-07-26T06:51:47.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:51:48.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:51:48.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:51:48.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672983 -lt 42949672983 2021-07-26T06:51:48.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:48.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509462 2021-07-26T06:51:48.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:48.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:51:48.613 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509462 2021-07-26T06:51:48.613 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:48.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509462 2021-07-26T06:51:48.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2021-07-26T06:51:48.614 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509462 2021-07-26T06:51:48.615 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:51:48.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509462 -lt 64424509462 2021-07-26T06:51:48.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:48.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345940 2021-07-26T06:51:48.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:48.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:51:48.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345940 2021-07-26T06:51:48.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:48.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345940 2021-07-26T06:51:48.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345940' 2021-07-26T06:51:48.886 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.3 seq 85899345940 2021-07-26T06:51:48.886 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:51:49.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345940 -lt 85899345940 2021-07-26T06:51:49.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:49.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182417 2021-07-26T06:51:49.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:49.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:51:49.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182417 2021-07-26T06:51:49.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:49.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182417 2021-07-26T06:51:49.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182417' 2021-07-26T06:51:49.188 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.4 seq 107374182417 2021-07-26T06:51:49.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:51:49.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182417 -lt 107374182417 2021-07-26T06:51:49.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:49.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018894 2021-07-26T06:51:49.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:49.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:51:49.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018894 2021-07-26T06:51:49.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:49.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018894 2021-07-26T06:51:49.458 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.5 seq 128849018894 2021-07-26T06:51:49.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018894' 2021-07-26T06:51:49.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:51:49.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018895 -lt 128849018894 2021-07-26T06:51:49.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:49.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855372 2021-07-26T06:51:49.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:49.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:51:49.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855372 2021-07-26T06:51:49.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:49.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855372 2021-07-26T06:51:49.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855372' 2021-07-26T06:51:49.774 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.6 seq 150323855372 2021-07-26T06:51:49.775 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:51:50.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855373 -lt 150323855372 2021-07-26T06:51:50.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:50.078 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691851 2021-07-26T06:51:50.078 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:50.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:51:50.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691851 2021-07-26T06:51:50.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:50.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691851 2021-07-26T06:51:50.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691851' 2021-07-26T06:51:50.081 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.7 seq 171798691851 2021-07-26T06:51:50.081 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:51:50.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691851 -lt 171798691851 2021-07-26T06:51:50.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:50.354 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528328 2021-07-26T06:51:50.354 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:50.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:51:50.355 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528328 2021-07-26T06:51:50.356 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:50.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528328 2021-07-26T06:51:50.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528328' 2021-07-26T06:51:50.357 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.8 seq 193273528328 2021-07-26T06:51:50.357 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:51:50.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528328 -lt 193273528328 2021-07-26T06:51:50.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:51:50.626 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-214748364806 2021-07-26T06:51:50.626 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:51:50.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:51:50.627 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-214748364806 2021-07-26T06:51:50.627 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:51:50.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=214748364806 2021-07-26T06:51:50.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 214748364806' 2021-07-26T06:51:50.628 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.9 seq 214748364806 2021-07-26T06:51:50.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:51:50.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 214748364807 -lt 214748364806 2021-07-26T06:51:50.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:51:50.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:51:50.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:51:51.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:51:51.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:51:51.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:51:51.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:51:51.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:51:51.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:51:51.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:51:51.272 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:51:51.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:51:51.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:51:51.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:51:51.533 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:51:51.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:51:51.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:51:51.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:51:51.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T06:51:51.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:843: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2021-07-26T06:51:51.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2021-07-26T06:51:51.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local poolname=ecpool 2021-07-26T06:51:51.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:250: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2021-07-26T06:51:51.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:51:51.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T06:51:51.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2021-07-26T06:51:51.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:51:51.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:51:51.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:51:52.302 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:51:52.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:51:52.982 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:51:52.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:51:52.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:51:52.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T06:51:53.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2021-07-26T06:51:53.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=ecpool 2021-07-26T06:51:53.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:51:53.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:51:53.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:51:53.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: local primary=3 2021-07-26T06:51:53.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2021-07-26T06:51:53.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2021-07-26T06:51:53.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=ecpool 2021-07-26T06:51:53.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=SOMETHING 2021-07-26T06:51:53.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: sed -e s/3// 2021-07-26T06:51:53.677 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:51:53.677 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T06:51:53.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=3 2021-07-26T06:51:53.947 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:51:53.948 INFO:tasks.workunit.client.0.smithi071.stderr:9 2021-07-26T06:51:53.948 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:51:53.948 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:51:53.948 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:51:53.948 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:51:53.949 INFO:tasks.workunit.client.0.smithi071.stderr:7' 2021-07-26T06:51:53.949 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 3 5 9 0 6 2 1 7 2021-07-26T06:51:53.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local -a osds 2021-07-26T06:51:53.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local not_primary_first=5 2021-07-26T06:51:53.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_second=9 2021-07-26T06:51:53.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2021-07-26T06:51:53.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2021-07-26T06:51:53.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2021-07-26T06:51:53.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=3 2021-07-26T06:51:53.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:51:53.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:51:53.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:51:53.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:51:53.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:51:53.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:51:53.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:51:53.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:51:53.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:51:53.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:51:53.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:51:53.956 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:51:53.956 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:51:53.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:51:53.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:51:53.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:51:54.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:51:54.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:51:54.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:51:54.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:51:54.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:51:54.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:51:54.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:51:54.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2021-07-26T06:51:56.067 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:51:56.067 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2021-07-26T06:51:56.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:51:56.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:51:56.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:51:56.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:51:56.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:51:56.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:51:56.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:51:56.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:51:56.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:51:56.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:51:56.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:51:56.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:51:56.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:51:56.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:51:56.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:51:56.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:51:56.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:51:56.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:51:56.593 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:51:56.593 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:51:56.593 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:51:56.593 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:51:56.593 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:51:56.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:51:56.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:51:56.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:51:56.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:51:56.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:51:56.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:51:56.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:51:56.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:51:56.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:51:56.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:51:56.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:51:56.596 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.3 2021-07-26T06:51:56.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:51:56.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:51:56.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:51:56.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:51:56.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:51:56.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:51:56.613 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:56.614+0000 7f4c922f60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:56.615 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:56.616+0000 7f4c922f60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:56.616 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:56.617+0000 7f4c922f60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:51:56.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:51:56.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:51:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:51:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:51:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:51:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:51:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:51:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:51:57.133 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:57.134+0000 7f4c922f60c0 -1 Falling back to public interface 2021-07-26T06:51:57.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:51:58.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:51:58.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:58.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:51:58.137 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:51:58.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:58.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:51:58.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:51:59.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:51:59.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:51:59.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:51:59.416 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:51:59.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:51:59.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:51:59.536 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:51:59.537+0000 7f4c922f60c0 -1 osd.3 67 log_to_monitors {default=true} 2021-07-26T06:51:59.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:52:00.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:52:00.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:52:00.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:52:00.801 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:52:00.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:52:00.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:52:01.247 INFO:tasks.workunit.client.0.smithi071.stderr:osd.3 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [20,67) [v2:127.0.0.1:6826/272444,v1:127.0.0.1:6827/272444] [v2:127.0.0.1:6828/272444,v1:127.0.0.1:6829/272444] exists,up 9c55da62-886d-4a69-a717-2a90c8e238a9 2021-07-26T06:52:01.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:52:01.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:52:01.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:52:01.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:52:01.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:52:01.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:52:01.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:52:01.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:52:01.252 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:52:01.253 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:52:01.253 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:52:01.254 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:52:01.254 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:52:01.254 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:52:01.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:52:01.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:52:01.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:52:01.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:52:01.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:52:01.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:52:01.681 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:52:01.682 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:52:01.682 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:52:01.682 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:52:01.682 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:52:01.683 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:52:01.683 INFO:tasks.workunit.client.0.smithi071.stderr:7 2021-07-26T06:52:01.683 INFO:tasks.workunit.client.0.smithi071.stderr:8 2021-07-26T06:52:01.683 INFO:tasks.workunit.client.0.smithi071.stderr:9' 2021-07-26T06:52:01.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:52:01.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:01.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:52:01.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836510 2021-07-26T06:52:01.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836510 2021-07-26T06:52:01.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510' 2021-07-26T06:52:01.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:01.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:52:01.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672987 2021-07-26T06:52:01.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672987 2021-07-26T06:52:01.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510 1-42949672987' 2021-07-26T06:52:01.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:01.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:52:02.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509466 2021-07-26T06:52:02.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509466 2021-07-26T06:52:02.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510 1-42949672987 2-64424509466' 2021-07-26T06:52:02.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:02.069 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:52:02.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=304942678018 2021-07-26T06:52:02.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 304942678018 2021-07-26T06:52:02.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510 1-42949672987 2-64424509466 3-304942678018' 2021-07-26T06:52:02.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:02.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:52:02.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182421 2021-07-26T06:52:02.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182421 2021-07-26T06:52:02.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510 1-42949672987 2-64424509466 3-304942678018 4-107374182421' 2021-07-26T06:52:02.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:02.308 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:52:02.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018899 2021-07-26T06:52:02.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018899 2021-07-26T06:52:02.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510 1-42949672987 2-64424509466 3-304942678018 4-107374182421 5-128849018899' 2021-07-26T06:52:02.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:02.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:52:02.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855376 2021-07-26T06:52:02.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855376 2021-07-26T06:52:02.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510 1-42949672987 2-64424509466 3-304942678018 4-107374182421 5-128849018899 6-150323855376' 2021-07-26T06:52:02.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:02.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:52:02.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691855 2021-07-26T06:52:02.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691855 2021-07-26T06:52:02.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510 1-42949672987 2-64424509466 3-304942678018 4-107374182421 5-128849018899 6-150323855376 7-171798691855' 2021-07-26T06:52:02.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:02.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:52:02.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528332 2021-07-26T06:52:02.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528332 2021-07-26T06:52:02.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510 1-42949672987 2-64424509466 3-304942678018 4-107374182421 5-128849018899 6-150323855376 7-171798691855 8-193273528332' 2021-07-26T06:52:02.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:02.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:52:02.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=214748364811 2021-07-26T06:52:02.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 214748364811 2021-07-26T06:52:02.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836510 1-42949672987 2-64424509466 3-304942678018 4-107374182421 5-128849018899 6-150323855376 7-171798691855 8-193273528332 9-214748364811' 2021-07-26T06:52:02.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:02.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836510 2021-07-26T06:52:02.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:02.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:52:02.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836510 2021-07-26T06:52:02.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:02.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836510 2021-07-26T06:52:02.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2021-07-26T06:52:02.904 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836510 2021-07-26T06:52:02.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:52:03.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836508 -lt 21474836510 2021-07-26T06:52:03.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:52:04.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:52:04.172 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:52:04.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836510 -lt 21474836510 2021-07-26T06:52:04.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:04.445 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672987 2021-07-26T06:52:04.445 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:04.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:52:04.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672987 2021-07-26T06:52:04.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:04.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672987 2021-07-26T06:52:04.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2021-07-26T06:52:04.448 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672987 2021-07-26T06:52:04.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:52:04.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672988 -lt 42949672987 2021-07-26T06:52:04.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:04.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509466 2021-07-26T06:52:04.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:04.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:52:04.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509466 2021-07-26T06:52:04.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:04.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509466 2021-07-26T06:52:04.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2021-07-26T06:52:04.754 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509466 2021-07-26T06:52:04.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:52:05.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509467 -lt 64424509466 2021-07-26T06:52:05.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:05.023 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-304942678018 2021-07-26T06:52:05.023 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:05.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:52:05.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-304942678018 2021-07-26T06:52:05.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:05.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=304942678018 2021-07-26T06:52:05.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 304942678018' 2021-07-26T06:52:05.026 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 304942678018 2021-07-26T06:52:05.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:52:05.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 304942678018 -lt 304942678018 2021-07-26T06:52:05.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:05.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182421 2021-07-26T06:52:05.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:05.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:52:05.488 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182421 2021-07-26T06:52:05.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:05.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182421 2021-07-26T06:52:05.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182421' 2021-07-26T06:52:05.490 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.4 seq 107374182421 2021-07-26T06:52:05.490 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:52:05.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182421 -lt 107374182421 2021-07-26T06:52:05.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:05.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018899 2021-07-26T06:52:05.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:05.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:52:05.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018899 2021-07-26T06:52:05.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:05.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018899 2021-07-26T06:52:05.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018899' 2021-07-26T06:52:05.760 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.5 seq 128849018899 2021-07-26T06:52:05.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:52:06.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018899 -lt 128849018899 2021-07-26T06:52:06.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:06.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855376 2021-07-26T06:52:06.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:06.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:52:06.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855376 2021-07-26T06:52:06.064 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:06.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855376 2021-07-26T06:52:06.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855376' 2021-07-26T06:52:06.065 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.6 seq 150323855376 2021-07-26T06:52:06.065 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:52:06.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855377 -lt 150323855376 2021-07-26T06:52:06.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:06.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691855 2021-07-26T06:52:06.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:06.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:52:06.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691855 2021-07-26T06:52:06.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:06.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691855 2021-07-26T06:52:06.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691855' 2021-07-26T06:52:06.337 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.7 seq 171798691855 2021-07-26T06:52:06.337 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:52:06.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691856 -lt 171798691855 2021-07-26T06:52:06.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:06.644 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528332 2021-07-26T06:52:06.645 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:06.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:52:06.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528332 2021-07-26T06:52:06.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:06.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528332 2021-07-26T06:52:06.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528332' 2021-07-26T06:52:06.647 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.8 seq 193273528332 2021-07-26T06:52:06.647 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:52:06.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528333 -lt 193273528332 2021-07-26T06:52:06.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:06.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-214748364811 2021-07-26T06:52:06.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:06.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:52:06.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-214748364811 2021-07-26T06:52:06.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:06.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=214748364811 2021-07-26T06:52:06.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 214748364811' 2021-07-26T06:52:06.961 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.9 seq 214748364811 2021-07-26T06:52:06.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:52:07.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 214748364811 -lt 214748364811 2021-07-26T06:52:07.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:52:07.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:52:07.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:52:07.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:52:07.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:52:07.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:52:07.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:52:07.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:52:07.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:52:07.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:52:07.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:52:07.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:52:07.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:52:07.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:52:07.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:52:08.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:52:08.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:52:08.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:52:08.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2021-07-26T06:52:08.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:52:08.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:52:08.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:52:08.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:52:08.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2021-07-26T06:52:08.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2021-07-26T06:52:08.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:52:08.461 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:52:08.461 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:08.461 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:08.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:08.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:08.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:51:43.118680+0000 2021-07-26T06:52:08.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:52:08.977 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:52:08.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:51:43.118680+0000 2021-07-26T06:52:08.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:52:08.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:51:43.118680+0000 2021-07-26T06:52:08.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:52:08.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:52:08.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:08.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:08.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:08.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:08.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:08.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:09.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:51:43.118680+0000 '>' 2021-07-26T06:51:43.118680+0000 2021-07-26T06:52:09.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:52:10.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:52:10.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:10.252 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:10.253 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:10.253 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:10.253 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:10.253 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:10.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:51:43.118680+0000 '>' 2021-07-26T06:51:43.118680+0000 2021-07-26T06:52:10.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:52:11.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:52:11.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:11.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:11.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:11.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:11.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:11.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:11.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:51:43.118680+0000 '>' 2021-07-26T06:51:43.118680+0000 2021-07-26T06:52:11.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:52:12.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:52:12.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:12.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:12.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:12.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:12.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:12.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:13.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:51:43.118680+0000 '>' 2021-07-26T06:51:43.118680+0000 2021-07-26T06:52:13.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:52:14.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:52:14.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:14.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:14.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:14.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:14.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:14.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:14.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:51:43.118680+0000 '>' 2021-07-26T06:51:43.118680+0000 2021-07-26T06:52:14.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:52:15.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:52:15.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:15.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:15.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:15.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:15.351 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:15.351 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:15.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:51:43.118680+0000 '>' 2021-07-26T06:51:43.118680+0000 2021-07-26T06:52:15.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:52:16.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:52:16.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:16.626 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:16.626 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:16.626 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:16.626 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:16.627 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:16.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:52:10.589542+0000 '>' 2021-07-26T06:51:43.118680+0000 2021-07-26T06:52:16.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:52:16.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:52:16.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:52:16.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:52:16.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=9 2021-07-26T06:52:16.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:52:16.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:52:16.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:52:16.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:52:16.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=9 2021-07-26T06:52:16.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:52:16.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2021-07-26T06:52:16.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:52:16.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:52:16.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:52:16.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:52:16.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:52:17.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:52:17.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:52:17.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:52:17.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:52:17.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=9 2021-07-26T06:52:17.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:52:17.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:52:17.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2021-07-26T06:52:18.900 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:52:18.901 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T06:52:18.901 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:52:18.901 INFO:tasks.workunit.client.0.smithi071.stdout:hinfo_key 2021-07-26T06:52:18.901 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:52:19.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2021-07-26T06:52:19.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:52:19.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:52:19.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=9 2021-07-26T06:52:19.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:52:19.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:52:19.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:52:19.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:52:19.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:52:19.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:52:19.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2021-07-26T06:52:19.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2021-07-26T06:52:19.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:52:19.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:52:19.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:52:19.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:52:19.181 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:52:19.182 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:52:19.182 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:52:19.182 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:52:19.183 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:52:19.183 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:52:19.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:52:19.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:52:19.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:52:19.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:52:19.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:52:19.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:52:19.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:52:19.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:52:19.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:52:19.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:52:19.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/9 2021-07-26T06:52:19.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.9 2021-07-26T06:52:19.186 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.9 2021-07-26T06:52:19.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 9 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:52:19.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/9/whoami 2021-07-26T06:52:19.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 9 = 9 ']' 2021-07-26T06:52:19.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:52:19.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:52:19.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:52:19.201 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:19.201+0000 7f8ad23fa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:52:19.203 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:19.203+0000 7f8ad23fa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:52:19.204 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:19.204+0000 7f8ad23fa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:52:19.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 9 2021-07-26T06:52:19.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:52:19.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=9 2021-07-26T06:52:19.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:52:19.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:52:19.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:52:19.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:52:19.458 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:52:19.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:52:19.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:52:19.720 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:19.721+0000 7f8ad23fa0c0 -1 Falling back to public interface 2021-07-26T06:52:19.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:52:20.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:52:20.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:52:20.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:52:20.783 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:52:20.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:52:20.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:52:21.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:52:22.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:52:22.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:52:22.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:52:22.056 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:52:22.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:52:22.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:52:22.147 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:22.148+0000 7f8ad23fa0c0 -1 osd.9 72 log_to_monitors {default=true} 2021-07-26T06:52:22.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:52:23.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:52:23.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:52:23.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:52:23.345 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:52:23.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:52:23.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:52:23.731 INFO:tasks.workunit.client.0.smithi071.stderr:osd.9 up in weight 1 up_from 76 up_thru 0 down_at 73 last_clean_interval [50,72) [v2:127.0.0.1:6874/274418,v1:127.0.0.1:6875/274418] [v2:127.0.0.1:6876/274418,v1:127.0.0.1:6877/274418] exists,up 1b1672b3-c8a7-4987-b315-589891f1b185 2021-07-26T06:52:23.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:52:23.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:52:23.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:52:23.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:52:23.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:52:23.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:52:23.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:52:23.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:52:23.733 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:52:23.733 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:52:23.733 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:52:23.734 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:52:23.734 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:52:23.734 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:52:23.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:52:23.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:52:23.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:52:23.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:52:23.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:52:24.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:52:24.161 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:52:24.161 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:52:24.161 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:52:24.161 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:52:24.162 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:52:24.162 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:52:24.162 INFO:tasks.workunit.client.0.smithi071.stderr:7 2021-07-26T06:52:24.162 INFO:tasks.workunit.client.0.smithi071.stderr:8 2021-07-26T06:52:24.163 INFO:tasks.workunit.client.0.smithi071.stderr:9' 2021-07-26T06:52:24.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:52:24.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:24.164 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:52:24.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836515 2021-07-26T06:52:24.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836515 2021-07-26T06:52:24.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515' 2021-07-26T06:52:24.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:24.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:52:24.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672993 2021-07-26T06:52:24.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672993 2021-07-26T06:52:24.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672993' 2021-07-26T06:52:24.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:24.421 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:52:24.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509472 2021-07-26T06:52:24.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509472 2021-07-26T06:52:24.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672993 2-64424509472' 2021-07-26T06:52:24.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:24.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:52:24.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=304942678023 2021-07-26T06:52:24.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 304942678023 2021-07-26T06:52:24.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672993 2-64424509472 3-304942678023' 2021-07-26T06:52:24.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:24.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:52:24.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182426 2021-07-26T06:52:24.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182426 2021-07-26T06:52:24.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672993 2-64424509472 3-304942678023 4-107374182426' 2021-07-26T06:52:24.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:24.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:52:24.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018904 2021-07-26T06:52:24.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018904 2021-07-26T06:52:24.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672993 2-64424509472 3-304942678023 4-107374182426 5-128849018904' 2021-07-26T06:52:24.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:24.898 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:52:25.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855382 2021-07-26T06:52:25.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855382 2021-07-26T06:52:25.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672993 2-64424509472 3-304942678023 4-107374182426 5-128849018904 6-150323855382' 2021-07-26T06:52:25.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:25.018 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:52:25.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691861 2021-07-26T06:52:25.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691861 2021-07-26T06:52:25.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672993 2-64424509472 3-304942678023 4-107374182426 5-128849018904 6-150323855382 7-171798691861' 2021-07-26T06:52:25.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:25.137 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:52:25.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528337 2021-07-26T06:52:25.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528337 2021-07-26T06:52:25.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672993 2-64424509472 3-304942678023 4-107374182426 5-128849018904 6-150323855382 7-171798691861 8-193273528337' 2021-07-26T06:52:25.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:25.254 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:52:25.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=326417514498 2021-07-26T06:52:25.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 326417514498 2021-07-26T06:52:25.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836515 1-42949672993 2-64424509472 3-304942678023 4-107374182426 5-128849018904 6-150323855382 7-171798691861 8-193273528337 9-326417514498' 2021-07-26T06:52:25.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:25.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836515 2021-07-26T06:52:25.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:25.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:52:25.374 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836515 2021-07-26T06:52:25.374 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:25.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836515 2021-07-26T06:52:25.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2021-07-26T06:52:25.375 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836515 2021-07-26T06:52:25.375 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:52:25.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836514 -lt 21474836515 2021-07-26T06:52:25.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:52:26.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:52:26.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:52:26.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836515 -lt 21474836515 2021-07-26T06:52:26.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:26.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672993 2021-07-26T06:52:26.920 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:26.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:52:26.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672993 2021-07-26T06:52:26.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:26.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672993 2021-07-26T06:52:26.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2021-07-26T06:52:26.922 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672993 2021-07-26T06:52:26.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:52:27.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672993 -lt 42949672993 2021-07-26T06:52:27.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:27.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509472 2021-07-26T06:52:27.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:27.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:52:27.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509472 2021-07-26T06:52:27.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:27.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509472 2021-07-26T06:52:27.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2021-07-26T06:52:27.237 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509472 2021-07-26T06:52:27.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:52:27.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509472 -lt 64424509472 2021-07-26T06:52:27.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:27.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-304942678023 2021-07-26T06:52:27.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:27.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:52:27.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-304942678023 2021-07-26T06:52:27.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:27.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=304942678023 2021-07-26T06:52:27.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 304942678023' 2021-07-26T06:52:27.544 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 304942678023 2021-07-26T06:52:27.544 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:52:27.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 304942678024 -lt 304942678023 2021-07-26T06:52:27.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:27.851 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182426 2021-07-26T06:52:27.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:27.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:52:27.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182426 2021-07-26T06:52:27.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:27.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182426 2021-07-26T06:52:27.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182426' 2021-07-26T06:52:27.854 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.4 seq 107374182426 2021-07-26T06:52:27.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:52:28.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182427 -lt 107374182426 2021-07-26T06:52:28.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:28.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018904 2021-07-26T06:52:28.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:28.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:52:28.171 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018904 2021-07-26T06:52:28.172 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:28.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018904 2021-07-26T06:52:28.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018904' 2021-07-26T06:52:28.172 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.5 seq 128849018904 2021-07-26T06:52:28.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:52:28.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018905 -lt 128849018904 2021-07-26T06:52:28.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:28.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855382 2021-07-26T06:52:28.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:28.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:52:28.445 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855382 2021-07-26T06:52:28.445 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:28.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855382 2021-07-26T06:52:28.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855382' 2021-07-26T06:52:28.447 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.6 seq 150323855382 2021-07-26T06:52:28.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:52:28.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855382 -lt 150323855382 2021-07-26T06:52:28.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:28.715 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691861 2021-07-26T06:52:28.715 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:28.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:52:28.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691861 2021-07-26T06:52:28.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:28.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691861 2021-07-26T06:52:28.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691861' 2021-07-26T06:52:28.718 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.7 seq 171798691861 2021-07-26T06:52:28.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:52:29.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691861 -lt 171798691861 2021-07-26T06:52:29.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:29.023 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528337 2021-07-26T06:52:29.023 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:29.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:52:29.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528337 2021-07-26T06:52:29.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:29.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528337 2021-07-26T06:52:29.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528337' 2021-07-26T06:52:29.025 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.8 seq 193273528337 2021-07-26T06:52:29.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:52:29.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528338 -lt 193273528337 2021-07-26T06:52:29.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:29.332 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-326417514498 2021-07-26T06:52:29.332 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:29.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:52:29.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-326417514498 2021-07-26T06:52:29.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:29.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=326417514498 2021-07-26T06:52:29.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 326417514498' 2021-07-26T06:52:29.335 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.9 seq 326417514498 2021-07-26T06:52:29.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:52:29.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 326417514499 -lt 326417514498 2021-07-26T06:52:29.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:52:29.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:52:29.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:52:30.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:52:30.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:52:30.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:52:30.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:52:30.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:52:30.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:52:30.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:52:30.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:52:30.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:52:30.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:52:30.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:52:30.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:52:30.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:52:30.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:52:30.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:52:30.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:52:30.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:52:30.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2021-07-26T06:52:30.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2021-07-26T06:52:30.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2021-07-26T06:52:30.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=5 2021-07-26T06:52:30.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:52:30.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:52:30.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:52:30.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=5 2021-07-26T06:52:30.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:52:30.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:52:30.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:52:30.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:52:30.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=5 2021-07-26T06:52:30.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:52:30.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2021-07-26T06:52:30.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:52:30.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:52:30.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:52:30.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:52:30.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:52:30.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:52:30.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:52:30.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:52:30.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:52:30.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=5 2021-07-26T06:52:30.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:52:30.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:52:30.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2021-07-26T06:52:32.773 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:52:32.773 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:52:32.774 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:52:33.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2021-07-26T06:52:33.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:52:33.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:52:33.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=5 2021-07-26T06:52:33.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:52:33.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:52:33.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:52:33.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:52:33.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:52:33.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:52:33.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2021-07-26T06:52:33.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2021-07-26T06:52:33.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:52:33.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:52:33.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:52:33.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:52:33.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:52:33.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:52:33.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:52:33.301 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:52:33.301 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:52:33.301 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:52:33.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:52:33.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:52:33.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:52:33.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:52:33.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:52:33.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:52:33.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:52:33.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:52:33.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:52:33.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:52:33.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/5 2021-07-26T06:52:33.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.5 2021-07-26T06:52:33.304 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.5 2021-07-26T06:52:33.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:52:33.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/5/whoami 2021-07-26T06:52:33.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 5 = 5 ']' 2021-07-26T06:52:33.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:52:33.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:52:33.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:52:33.320 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:33.319+0000 7fbaafcf40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:52:33.322 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:33.321+0000 7fbaafcf40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:52:33.323 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:33.323+0000 7fbaafcf40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:52:33.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 5 2021-07-26T06:52:33.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:52:33.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:52:33.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:52:33.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:52:33.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:52:33.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:52:33.580 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:52:33.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:52:33.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:52:33.839 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:33.838+0000 7fbaafcf40c0 -1 Falling back to public interface 2021-07-26T06:52:33.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:52:34.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:52:34.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:52:34.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:52:34.851 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:52:34.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:52:34.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:52:35.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:52:36.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:52:36.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:52:36.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:52:36.131 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:52:36.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:52:36.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:52:36.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:52:36.732 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:36.732+0000 7fbaafcf40c0 -1 osd.5 78 log_to_monitors {default=true} 2021-07-26T06:52:37.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:52:37.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:52:37.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:52:37.620 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:52:37.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:52:37.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:52:38.115 INFO:tasks.workunit.client.0.smithi071.stderr:osd.5 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [30,78) [v2:127.0.0.1:6842/276155,v1:127.0.0.1:6843/276155] [v2:127.0.0.1:6844/276155,v1:127.0.0.1:6845/276155] exists,up bbee9dc4-1bba-496c-96c5-7f58e2aee6bc 2021-07-26T06:52:38.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:52:38.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:52:38.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:52:38.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:52:38.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:52:38.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:52:38.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:52:38.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:52:38.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:52:38.117 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:52:38.118 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:52:38.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:52:38.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:52:38.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:52:38.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:52:38.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:52:38.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:52:38.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:52:38.210 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:52:38.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:52:38.575 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:52:38.575 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:52:38.575 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:52:38.576 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:52:38.576 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:52:38.576 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:52:38.576 INFO:tasks.workunit.client.0.smithi071.stderr:7 2021-07-26T06:52:38.577 INFO:tasks.workunit.client.0.smithi071.stderr:8 2021-07-26T06:52:38.577 INFO:tasks.workunit.client.0.smithi071.stderr:9' 2021-07-26T06:52:38.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:52:38.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:38.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:52:38.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836519 2021-07-26T06:52:38.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836519 2021-07-26T06:52:38.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519' 2021-07-26T06:52:38.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:38.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:52:38.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672997 2021-07-26T06:52:38.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672997 2021-07-26T06:52:38.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519 1-42949672997' 2021-07-26T06:52:38.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:38.930 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:52:39.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509476 2021-07-26T06:52:39.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509476 2021-07-26T06:52:39.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519 1-42949672997 2-64424509476' 2021-07-26T06:52:39.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:39.048 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:52:39.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=304942678027 2021-07-26T06:52:39.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 304942678027 2021-07-26T06:52:39.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519 1-42949672997 2-64424509476 3-304942678027' 2021-07-26T06:52:39.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:39.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:52:39.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182430 2021-07-26T06:52:39.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182430 2021-07-26T06:52:39.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519 1-42949672997 2-64424509476 3-304942678027 4-107374182430' 2021-07-26T06:52:39.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:39.286 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:52:39.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=352187318274 2021-07-26T06:52:39.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 352187318274 2021-07-26T06:52:39.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519 1-42949672997 2-64424509476 3-304942678027 4-107374182430 5-352187318274' 2021-07-26T06:52:39.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:39.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:52:39.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855386 2021-07-26T06:52:39.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855386 2021-07-26T06:52:39.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519 1-42949672997 2-64424509476 3-304942678027 4-107374182430 5-352187318274 6-150323855386' 2021-07-26T06:52:39.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:39.521 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:52:39.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691864 2021-07-26T06:52:39.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691864 2021-07-26T06:52:39.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519 1-42949672997 2-64424509476 3-304942678027 4-107374182430 5-352187318274 6-150323855386 7-171798691864' 2021-07-26T06:52:39.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:39.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:52:39.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528341 2021-07-26T06:52:39.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528341 2021-07-26T06:52:39.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519 1-42949672997 2-64424509476 3-304942678027 4-107374182430 5-352187318274 6-150323855386 7-171798691864 8-193273528341' 2021-07-26T06:52:39.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:52:39.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:52:39.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=326417514502 2021-07-26T06:52:39.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 326417514502 2021-07-26T06:52:39.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836519 1-42949672997 2-64424509476 3-304942678027 4-107374182430 5-352187318274 6-150323855386 7-171798691864 8-193273528341 9-326417514502' 2021-07-26T06:52:39.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:39.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836519 2021-07-26T06:52:39.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:39.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:52:39.915 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836519 2021-07-26T06:52:39.915 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:39.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836519 2021-07-26T06:52:39.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2021-07-26T06:52:39.916 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836519 2021-07-26T06:52:39.916 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:52:40.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836519 -lt 21474836519 2021-07-26T06:52:40.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:40.182 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672997 2021-07-26T06:52:40.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:40.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:52:40.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672997 2021-07-26T06:52:40.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:40.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672997 2021-07-26T06:52:40.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2021-07-26T06:52:40.186 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672997 2021-07-26T06:52:40.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:52:40.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672997 -lt 42949672997 2021-07-26T06:52:40.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:40.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509476 2021-07-26T06:52:40.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:40.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:52:40.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509476 2021-07-26T06:52:40.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:40.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509476 2021-07-26T06:52:40.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2021-07-26T06:52:40.457 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509476 2021-07-26T06:52:40.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:52:40.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509476 -lt 64424509476 2021-07-26T06:52:40.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:40.767 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-304942678027 2021-07-26T06:52:40.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:40.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:52:40.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-304942678027 2021-07-26T06:52:40.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:40.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=304942678027 2021-07-26T06:52:40.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 304942678027' 2021-07-26T06:52:40.770 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 304942678027 2021-07-26T06:52:40.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:52:41.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 304942678027 -lt 304942678027 2021-07-26T06:52:41.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:41.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182430 2021-07-26T06:52:41.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:41.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:52:41.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182430 2021-07-26T06:52:41.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:41.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182430 2021-07-26T06:52:41.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182430' 2021-07-26T06:52:41.082 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.4 seq 107374182430 2021-07-26T06:52:41.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:52:41.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182430 -lt 107374182430 2021-07-26T06:52:41.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:41.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-352187318274 2021-07-26T06:52:41.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:41.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:52:41.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-352187318274 2021-07-26T06:52:41.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:41.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=352187318274 2021-07-26T06:52:41.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 352187318274' 2021-07-26T06:52:41.394 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.5 seq 352187318274 2021-07-26T06:52:41.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:52:41.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 352187318274 -lt 352187318274 2021-07-26T06:52:41.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:41.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855386 2021-07-26T06:52:41.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:41.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:52:41.703 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855386 2021-07-26T06:52:41.703 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:41.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855386 2021-07-26T06:52:41.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855386' 2021-07-26T06:52:41.704 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.6 seq 150323855386 2021-07-26T06:52:41.704 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:52:41.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855386 -lt 150323855386 2021-07-26T06:52:41.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:41.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691864 2021-07-26T06:52:41.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:41.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:52:41.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691864 2021-07-26T06:52:41.976 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:41.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691864 2021-07-26T06:52:41.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691864' 2021-07-26T06:52:41.977 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.7 seq 171798691864 2021-07-26T06:52:41.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:52:42.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691865 -lt 171798691864 2021-07-26T06:52:42.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:42.285 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528341 2021-07-26T06:52:42.286 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:42.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:52:42.287 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528341 2021-07-26T06:52:42.287 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:42.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528341 2021-07-26T06:52:42.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528341' 2021-07-26T06:52:42.288 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.8 seq 193273528341 2021-07-26T06:52:42.288 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:52:42.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528342 -lt 193273528341 2021-07-26T06:52:42.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:52:42.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-326417514502 2021-07-26T06:52:42.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:52:42.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:52:42.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-326417514502 2021-07-26T06:52:42.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:52:42.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=326417514502 2021-07-26T06:52:42.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 326417514502' 2021-07-26T06:52:42.590 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.9 seq 326417514502 2021-07-26T06:52:42.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:52:42.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 326417514502 -lt 326417514502 2021-07-26T06:52:42.894 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:52:42.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:52:42.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:52:43.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:52:43.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:52:43.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:52:43.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:52:43.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:52:43.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:52:43.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:52:43.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:52:43.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:52:43.565 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:52:43.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:52:43.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:52:43.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:52:43.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:52:43.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:52:43.894 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2021-07-26T06:52:43.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:52:43.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:52:43.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:52:43.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:52:44.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2021-07-26T06:52:44.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2021-07-26T06:52:44.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:52:44.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:52:44.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:44.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:44.209 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:44.209 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:44.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:52:10.589542+0000 2021-07-26T06:52:44.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:52:44.762 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:52:44.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:52:10.589542+0000 2021-07-26T06:52:44.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:52:44.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:52:10.589542+0000 2021-07-26T06:52:44.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:52:44.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:52:44.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:44.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:44.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:44.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:44.775 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:44.775 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:45.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:52:10.589542+0000 '>' 2021-07-26T06:52:10.589542+0000 2021-07-26T06:52:45.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:52:46.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:52:46.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:46.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:46.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:46.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:46.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:46.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:46.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:52:10.589542+0000 '>' 2021-07-26T06:52:10.589542+0000 2021-07-26T06:52:46.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:52:47.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:52:47.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:47.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:47.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:47.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:47.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:47.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:47.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:52:10.589542+0000 '>' 2021-07-26T06:52:10.589542+0000 2021-07-26T06:52:47.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:52:48.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:52:48.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:48.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:48.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:48.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:48.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:48.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:48.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:52:10.589542+0000 '>' 2021-07-26T06:52:10.589542+0000 2021-07-26T06:52:48.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:52:49.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:52:49.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:49.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:49.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:49.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:49.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:49.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:50.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:52:10.589542+0000 '>' 2021-07-26T06:52:10.589542+0000 2021-07-26T06:52:50.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:52:51.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:52:51.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:51.138 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:51.138 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:51.138 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:51.138 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:51.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:51.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:52:10.589542+0000 '>' 2021-07-26T06:52:10.589542+0000 2021-07-26T06:52:51.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:52:52.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:52:52.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:52:52.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:52:52.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:52:52.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:52:52.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:52:52.416 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:52:52.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:52:46.610084+0000 '>' 2021-07-26T06:52:10.589542+0000 2021-07-26T06:52:52.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:52:52.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:52:52.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:52:52.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:52:52.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=9 2021-07-26T06:52:52.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:52:52.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:52:52.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:52:52.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:52:52.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=9 2021-07-26T06:52:52.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:52:52.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2021-07-26T06:52:52.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:52:52.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:52:52.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:52:52.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:52:52.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:52:52.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:52:52.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:52:52.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:52:52.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:52:52.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=9 2021-07-26T06:52:52.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:52:52.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:52:52.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2021-07-26T06:52:54.569 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:52:54.570 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T06:52:54.570 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:52:54.570 INFO:tasks.workunit.client.0.smithi071.stdout:hinfo_key 2021-07-26T06:52:54.571 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:52:55.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2021-07-26T06:52:55.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:52:55.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:52:55.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=9 2021-07-26T06:52:55.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:52:55.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:52:55.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:52:55.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:52:55.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:52:55.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:52:55.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2021-07-26T06:52:55.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2021-07-26T06:52:55.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:52:55.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:52:55.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:52:55.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:52:55.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:52:55.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:52:55.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:52:55.099 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:52:55.099 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:52:55.099 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:52:55.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:52:55.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:52:55.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:52:55.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:52:55.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:52:55.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:52:55.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:52:55.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:52:55.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:52:55.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:52:55.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/9 2021-07-26T06:52:55.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.9 2021-07-26T06:52:55.102 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.9 2021-07-26T06:52:55.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 9 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:52:55.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/9/whoami 2021-07-26T06:52:55.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 9 = 9 ']' 2021-07-26T06:52:55.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:52:55.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:52:55.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:52:55.118 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:55.117+0000 7efc63b8e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:52:55.120 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:55.119+0000 7efc63b8e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:52:55.121 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:55.120+0000 7efc63b8e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:52:55.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 9 2021-07-26T06:52:55.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:52:55.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=9 2021-07-26T06:52:55.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:52:55.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:52:55.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:52:55.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:52:55.376 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:52:55.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:52:55.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:52:55.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:52:55.644 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:55.643+0000 7efc63b8e0c0 -1 Falling back to public interface 2021-07-26T06:52:56.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:52:56.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:52:56.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:52:56.640 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:52:56.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:52:56.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:52:56.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:52:57.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:52:57.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:52:57.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:52:57.927 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:52:57.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:52:57.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:52:58.029 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:52:58.028+0000 7efc63b8e0c0 -1 osd.9 84 log_to_monitors {default=true} 2021-07-26T06:52:58.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:52:59.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:52:59.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:52:59.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:52:59.251 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:52:59.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:52:59.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:52:59.610 INFO:tasks.workunit.client.0.smithi071.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [76,84) [v2:127.0.0.1:6874/278097,v1:127.0.0.1:6875/278097] [v2:127.0.0.1:6876/278097,v1:127.0.0.1:6877/278097] exists,up 1b1672b3-c8a7-4987-b315-589891f1b185 2021-07-26T06:52:59.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:52:59.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:52:59.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:52:59.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:52:59.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:52:59.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:52:59.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:52:59.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:52:59.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:52:59.612 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:52:59.613 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:52:59.613 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:52:59.613 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:52:59.613 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:52:59.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:52:59.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:52:59.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:52:59.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:52:59.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:53:00.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:53:00.130 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:53:00.131 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:53:00.131 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:53:00.131 INFO:tasks.workunit.client.0.smithi071.stderr:4 2021-07-26T06:53:00.131 INFO:tasks.workunit.client.0.smithi071.stderr:5 2021-07-26T06:53:00.131 INFO:tasks.workunit.client.0.smithi071.stderr:6 2021-07-26T06:53:00.132 INFO:tasks.workunit.client.0.smithi071.stderr:7 2021-07-26T06:53:00.132 INFO:tasks.workunit.client.0.smithi071.stderr:8 2021-07-26T06:53:00.132 INFO:tasks.workunit.client.0.smithi071.stderr:9' 2021-07-26T06:53:00.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:53:00.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:00.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:53:00.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836524 2021-07-26T06:53:00.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836524 2021-07-26T06:53:00.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524' 2021-07-26T06:53:00.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:00.292 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:53:00.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673002 2021-07-26T06:53:00.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673002 2021-07-26T06:53:00.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673002' 2021-07-26T06:53:00.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:00.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:53:00.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509481 2021-07-26T06:53:00.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509481 2021-07-26T06:53:00.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509481' 2021-07-26T06:53:00.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:00.551 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:53:00.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=304942678033 2021-07-26T06:53:00.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 304942678033 2021-07-26T06:53:00.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509481 3-304942678033' 2021-07-26T06:53:00.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:00.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:53:00.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182435 2021-07-26T06:53:00.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182435 2021-07-26T06:53:00.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509481 3-304942678033 4-107374182435' 2021-07-26T06:53:00.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:00.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:53:00.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=352187318279 2021-07-26T06:53:00.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 352187318279 2021-07-26T06:53:00.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509481 3-304942678033 4-107374182435 5-352187318279' 2021-07-26T06:53:00.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:00.906 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:53:01.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855391 2021-07-26T06:53:01.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855391 2021-07-26T06:53:01.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509481 3-304942678033 4-107374182435 5-352187318279 6-150323855391' 2021-07-26T06:53:01.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:01.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:53:01.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691870 2021-07-26T06:53:01.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691870 2021-07-26T06:53:01.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509481 3-304942678033 4-107374182435 5-352187318279 6-150323855391 7-171798691870' 2021-07-26T06:53:01.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:01.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:53:01.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528347 2021-07-26T06:53:01.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528347 2021-07-26T06:53:01.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509481 3-304942678033 4-107374182435 5-352187318279 6-150323855391 7-171798691870 8-193273528347' 2021-07-26T06:53:01.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:01.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:53:01.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=377957122050 2021-07-26T06:53:01.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 377957122050 2021-07-26T06:53:01.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509481 3-304942678033 4-107374182435 5-352187318279 6-150323855391 7-171798691870 8-193273528347 9-377957122050' 2021-07-26T06:53:01.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:01.385 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836524 2021-07-26T06:53:01.385 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:01.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:53:01.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836524 2021-07-26T06:53:01.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:01.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836524 2021-07-26T06:53:01.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2021-07-26T06:53:01.388 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836524 2021-07-26T06:53:01.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:53:01.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836523 -lt 21474836524 2021-07-26T06:53:01.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:53:02.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:53:02.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:53:02.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836524 -lt 21474836524 2021-07-26T06:53:02.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:02.933 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673002 2021-07-26T06:53:02.933 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:02.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:53:02.934 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673002 2021-07-26T06:53:02.934 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:02.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673002 2021-07-26T06:53:02.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2021-07-26T06:53:02.936 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949673002 2021-07-26T06:53:02.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:53:03.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673002 -lt 42949673002 2021-07-26T06:53:03.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:03.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509481 2021-07-26T06:53:03.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:03.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:53:03.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509481 2021-07-26T06:53:03.238 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:03.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509481 2021-07-26T06:53:03.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2021-07-26T06:53:03.239 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509481 2021-07-26T06:53:03.239 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:53:03.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509481 -lt 64424509481 2021-07-26T06:53:03.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:03.508 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-304942678033 2021-07-26T06:53:03.508 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:03.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:53:03.509 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-304942678033 2021-07-26T06:53:03.509 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:03.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=304942678033 2021-07-26T06:53:03.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 304942678033' 2021-07-26T06:53:03.510 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.3 seq 304942678033 2021-07-26T06:53:03.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:53:03.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 304942678033 -lt 304942678033 2021-07-26T06:53:03.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:03.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182435 2021-07-26T06:53:03.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:03.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:53:03.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182435 2021-07-26T06:53:03.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:03.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182435 2021-07-26T06:53:03.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182435' 2021-07-26T06:53:03.826 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.4 seq 107374182435 2021-07-26T06:53:03.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:53:04.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182436 -lt 107374182435 2021-07-26T06:53:04.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:04.124 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-352187318279 2021-07-26T06:53:04.124 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:04.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:53:04.125 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-352187318279 2021-07-26T06:53:04.125 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:04.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=352187318279 2021-07-26T06:53:04.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 352187318279' 2021-07-26T06:53:04.127 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.5 seq 352187318279 2021-07-26T06:53:04.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:53:04.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 352187318280 -lt 352187318279 2021-07-26T06:53:04.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:04.420 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855391 2021-07-26T06:53:04.421 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:04.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:53:04.422 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855391 2021-07-26T06:53:04.422 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:04.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855391 2021-07-26T06:53:04.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855391' 2021-07-26T06:53:04.423 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.6 seq 150323855391 2021-07-26T06:53:04.423 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:53:04.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855392 -lt 150323855391 2021-07-26T06:53:04.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:04.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691870 2021-07-26T06:53:04.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:04.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:53:04.691 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691870 2021-07-26T06:53:04.691 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:04.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691870 2021-07-26T06:53:04.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691870' 2021-07-26T06:53:04.693 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.7 seq 171798691870 2021-07-26T06:53:04.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:53:04.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691870 -lt 171798691870 2021-07-26T06:53:04.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:04.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528347 2021-07-26T06:53:04.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:04.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:53:04.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528347 2021-07-26T06:53:04.992 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:04.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528347 2021-07-26T06:53:04.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528347' 2021-07-26T06:53:04.993 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.8 seq 193273528347 2021-07-26T06:53:04.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:53:05.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528347 -lt 193273528347 2021-07-26T06:53:05.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:05.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-377957122050 2021-07-26T06:53:05.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:05.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:53:05.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-377957122050 2021-07-26T06:53:05.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:05.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=377957122050 2021-07-26T06:53:05.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2021-07-26T06:53:05.302 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.9 seq 377957122050 2021-07-26T06:53:05.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:53:05.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 377957122051 -lt 377957122050 2021-07-26T06:53:05.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:53:05.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:05.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:05.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:53:05.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:53:05.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:53:05.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:53:05.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:53:05.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:53:05.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:53:05.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:53:06.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:53:06.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:53:06.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:06.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:06.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:53:06.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:53:06.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:53:06.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:53:06.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:53:06.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2021-07-26T06:53:06.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2021-07-26T06:53:06.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2021-07-26T06:53:06.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=5 2021-07-26T06:53:06.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=9 2021-07-26T06:53:06.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2021-07-26T06:53:06.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:53:06.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:53:06.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:53:06.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 279421"' 2021-07-26T06:53:06.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 279421' 2021-07-26T06:53:06.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2021-07-26T06:53:06.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:53:06.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:53:06.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 279423"' 2021-07-26T06:53:06.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 279423' 2021-07-26T06:53:06.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:53:06.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2021-07-26T06:53:06.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 279421 279423' 2021-07-26T06:53:06.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:53:06.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:53:06.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 279421 2021-07-26T06:53:06.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/279424: /' 2021-07-26T06:53:06.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2021-07-26T06:53:06.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/279426: /' 2021-07-26T06:53:09.617 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:53:09.618 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:53:09.618 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=5 2021-07-26T06:53:09.618 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:53:09.619 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:53:09.619 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:53:09.619 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:53:09.619 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=5 2021-07-26T06:53:09.620 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:53:09.620 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2021-07-26T06:53:09.620 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:53:09.620 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:53:09.621 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:53:09.621 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:53:09.621 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:53:09.621 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:53:09.622 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:53:09.622 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:53:09.622 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:53:09.622 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=5 2021-07-26T06:53:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:53:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:53:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2021-07-26T06:53:09.623 INFO:tasks.workunit.client.0.smithi071.stderr:279424: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:53:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:279424: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:53:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:279424: remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:53:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2021-07-26T06:53:09.624 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:53:09.625 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:53:09.625 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=5 2021-07-26T06:53:09.625 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:53:09.625 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:53:09.625 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:53:09.626 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:53:09.626 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:53:09.627 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:53:09.627 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2021-07-26T06:53:09.627 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2021-07-26T06:53:09.627 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:53:09.628 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:53:09.628 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:53:09.628 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:53:09.628 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:53:09.629 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:53:09.629 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:53:09.629 INFO:tasks.workunit.client.0.smithi071.stderr:279424: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:53:09.629 INFO:tasks.workunit.client.0.smithi071.stderr:279424: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:53:09.629 INFO:tasks.workunit.client.0.smithi071.stderr:279424: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:53:09.630 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:53:09.630 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:53:09.630 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:53:09.630 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:53:09.631 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:53:09.631 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:53:09.631 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:53:09.631 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:53:09.632 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:53:09.632 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:53:09.632 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/5 2021-07-26T06:53:09.632 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.5 2021-07-26T06:53:09.632 INFO:tasks.workunit.client.0.smithi071.stderr:279424: start osd.5 2021-07-26T06:53:09.729 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:53:09.729 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:53:09.729 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=9 2021-07-26T06:53:09.730 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:53:09.730 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2021-07-26T06:53:09.730 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:53:09.730 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:53:09.731 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=9 2021-07-26T06:53:09.731 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:53:09.731 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2021-07-26T06:53:09.731 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:53:09.731 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:53:09.732 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:53:09.732 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:53:09.732 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:53:09.732 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:53:09.733 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2021-07-26T06:53:09.733 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:53:09.733 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:53:09.733 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=9 2021-07-26T06:53:09.734 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:53:09.734 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:53:09.734 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2021-07-26T06:53:09.734 INFO:tasks.workunit.client.0.smithi071.stderr:279426: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:53:09.735 INFO:tasks.workunit.client.0.smithi071.stderr:279426: Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T06:53:09.735 INFO:tasks.workunit.client.0.smithi071.stderr:279426: remove 2#2:eb822e21:::SOMETHING:head# 2021-07-26T06:53:09.735 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2021-07-26T06:53:09.735 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:53:09.736 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:53:09.736 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=9 2021-07-26T06:53:09.736 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:53:09.736 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:53:09.737 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:53:09.737 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:53:09.737 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:53:09.738 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:53:09.738 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2021-07-26T06:53:09.738 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2021-07-26T06:53:09.738 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:53:09.739 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:53:09.739 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:53:09.739 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:53:09.739 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:53:09.740 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:53:09.740 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:53:09.740 INFO:tasks.workunit.client.0.smithi071.stderr:279426: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:53:09.740 INFO:tasks.workunit.client.0.smithi071.stderr:279426: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:53:09.740 INFO:tasks.workunit.client.0.smithi071.stderr:279426: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:53:09.741 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:53:09.741 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:53:09.741 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:53:09.741 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:53:09.742 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:53:09.742 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:53:09.742 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:53:09.742 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:53:09.743 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:53:09.743 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:53:09.743 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/9 2021-07-26T06:53:09.743 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.9 2021-07-26T06:53:09.744 INFO:tasks.workunit.client.0.smithi071.stderr:279426: start osd.9 2021-07-26T06:53:13.821 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 9 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:53:13.821 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/5/whoami 2021-07-26T06:53:13.822 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 5 = 5 ']' 2021-07-26T06:53:13.822 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:53:13.822 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:53:13.822 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:53:13.823 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 2021-07-26T06:53:09.644+0000 7f40485310c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:13.823 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 2021-07-26T06:53:09.646+0000 7f40485310c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:13.823 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 2021-07-26T06:53:09.647+0000 7f40485310c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:13.824 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 5 2021-07-26T06:53:13.824 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:53:13.824 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:53:13.824 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:53:13.825 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:53:13.825 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:13.825 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:53:13.825 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 0 2021-07-26T06:53:13.826 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:13.826 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:53:13.826 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 2021-07-26T06:53:10.171+0000 7f40485310c0 -1 Falling back to public interface 2021-07-26T06:53:13.826 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:13.827 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:13.827 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:13.827 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:53:13.827 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 1 2021-07-26T06:53:13.828 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:13.828 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:53:13.828 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:13.829 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:13.829 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:13.830 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:53:13.830 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 2 2021-07-26T06:53:13.830 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:13.830 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:53:13.831 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:13.831 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 2021-07-26T06:53:13.065+0000 7f40485310c0 -1 osd.5 90 log_to_monitors {default=true} 2021-07-26T06:53:13.831 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:13.832 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:13.832 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:53:13.833 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 3 2021-07-26T06:53:13.833 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:13.919 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clname.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:53:13.919 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/9/whoami 2021-07-26T06:53:13.919 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 9 = 9 ']' 2021-07-26T06:53:13.919 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:53:13.920 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:53:13.920 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:53:13.920 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 2021-07-26T06:53:09.753+0000 7f44ef3b90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:13.921 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 2021-07-26T06:53:09.755+0000 7f44ef3b90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:13.921 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 2021-07-26T06:53:09.756+0000 7f44ef3b90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:13.921 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 9 2021-07-26T06:53:13.921 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:53:13.921 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=9 2021-07-26T06:53:13.922 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:53:13.922 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:53:13.922 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:13.922 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:53:13.923 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 0 2021-07-26T06:53:13.923 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:13.923 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:53:13.923 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 2021-07-26T06:53:10.273+0000 7f44ef3b90c0 -1 Falling back to public interface 2021-07-26T06:53:13.923 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:13.924 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:13.924 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:13.924 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:53:13.924 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 1 2021-07-26T06:53:13.924 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:13.925 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:53:13.925 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:13.925 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:13.925 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:13.925 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:53:13.926 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 2 2021-07-26T06:53:13.926 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:13.927 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:53:13.927 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:13.927 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 2021-07-26T06:53:13.224+0000 7f44ef3b90c0 -1 osd.9 90 log_to_monitors {default=true} 2021-07-26T06:53:13.927 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:13.927 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:13.928 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:53:13.928 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 3 2021-07-26T06:53:13.928 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:15.205 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:53:15.205 INFO:tasks.workunit.client.0.smithi071.stderr:279426: osd.9 up in weight 1 up_from 95 up_thru 0 down_at 91 last_clean_interval [88,90) [v2:127.0.0.1:6874/280274,v1:127.0.0.1:6875/280274] [v2:127.0.0.1:6876/280274,v1:127.0.0.1:6877/280274] exists,up 1b1672b3-c8a7-4987-b315-589891f1b185 2021-07-26T06:53:15.206 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:53:15.206 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:53:15.206 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:53:15.206 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:53:15.206 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:53:15.207 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:53:15.207 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:53:15.207 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:53:15.208 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:53:15.208 INFO:tasks.workunit.client.0.smithi071.stderr:279426: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:53:15.208 INFO:tasks.workunit.client.0.smithi071.stderr:279426: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:53:15.208 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:53:15.208 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:53:15.209 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:53:15.209 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:53:15.209 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:53:15.209 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:53:15.210 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:53:15.210 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:53:15.210 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:53:15.210 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 1 2021-07-26T06:53:15.210 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 2 2021-07-26T06:53:15.211 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 3 2021-07-26T06:53:15.211 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 4 2021-07-26T06:53:15.211 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 5 2021-07-26T06:53:15.211 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 6 2021-07-26T06:53:15.211 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 7 2021-07-26T06:53:15.212 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 8 2021-07-26T06:53:15.212 INFO:tasks.workunit.client.0.smithi071.stderr:279426: 9' 2021-07-26T06:53:15.213 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:53:15.213 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:15.213 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:53:15.213 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836529 2021-07-26T06:53:15.214 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836529 2021-07-26T06:53:15.214 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529' 2021-07-26T06:53:15.214 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:15.214 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:53:15.215 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673006 2021-07-26T06:53:15.215 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673006 2021-07-26T06:53:15.215 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673006' 2021-07-26T06:53:15.215 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:53:15.216 INFO:tasks.workunit.client.0.smithi071.stderr:279424: osd.5 up in weight 1 up_from 95 up_thru 82 down_at 91 last_clean_interval [82,90) [v2:127.0.0.1:6842/280229,v1:127.0.0.1:6843/280229] [v2:127.0.0.1:6844/280229,v1:127.0.0.1:6845/280229] exists,up bbee9dc4-1bba-496c-96c5-7f58e2aee6bc 2021-07-26T06:53:15.216 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:53:15.216 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:53:15.216 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:53:15.216 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:53:15.217 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:53:15.217 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:53:15.217 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:53:15.217 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:53:15.218 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:53:15.218 INFO:tasks.workunit.client.0.smithi071.stderr:279424: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:53:15.218 INFO:tasks.workunit.client.0.smithi071.stderr:279424: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:53:15.218 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:53:15.218 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:53:15.219 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:53:15.219 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:53:15.219 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:53:15.219 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:53:15.220 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:53:15.220 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:53:15.220 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:53:15.220 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 1 2021-07-26T06:53:15.220 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 2 2021-07-26T06:53:15.221 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 3 2021-07-26T06:53:15.221 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 4 2021-07-26T06:53:15.221 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 5 2021-07-26T06:53:15.221 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 6 2021-07-26T06:53:15.222 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 7 2021-07-26T06:53:15.222 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 8 2021-07-26T06:53:15.222 INFO:tasks.workunit.client.0.smithi071.stderr:279424: 9' 2021-07-26T06:53:15.222 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:53:15.222 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:15.223 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:53:15.223 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836528 2021-07-26T06:53:15.223 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836528 2021-07-26T06:53:15.223 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528' 2021-07-26T06:53:15.224 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:15.224 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:53:15.224 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673007 2021-07-26T06:53:15.224 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673007 2021-07-26T06:53:15.224 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528 1-42949673007' 2021-07-26T06:53:16.400 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standaone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:16.400 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:53:16.400 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509485 2021-07-26T06:53:16.401 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509485 2021-07-26T06:53:16.401 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509485' 2021-07-26T06:53:16.401 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:16.401 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:53:16.402 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=304942678037 2021-07-26T06:53:16.402 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 304942678037 2021-07-26T06:53:16.402 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509485 3-304942678037' 2021-07-26T06:53:16.402 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:16.402 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:53:16.403 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182439 2021-07-26T06:53:16.403 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182439 2021-07-26T06:53:16.403 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509485 3-304942678037 4-107374182439' 2021-07-26T06:53:16.403 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:16.404 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:53:16.404 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=408021893123 2021-07-26T06:53:16.404 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 408021893123 2021-07-26T06:53:16.404 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509485 3-304942678037 4-107374182439 5-408021893123' 2021-07-26T06:53:16.404 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:16.405 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:53:16.405 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855395 2021-07-26T06:53:16.405 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855395 2021-07-26T06:53:16.405 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509485 3-304942678037 4-107374182439 5-408021893123 6-150323855395' 2021-07-26T06:53:16.406 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:16.406 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:53:16.406 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691874 2021-07-26T06:53:16.406 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691874 2021-07-26T06:53:16.406 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509485 3-304942678037 4-107374182439 5-408021893123 6-150323855395 7-171798691874' 2021-07-26T06:53:16.407 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:16.407 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalonlone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:16.407 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:53:16.407 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509486 2021-07-26T06:53:16.408 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509486 2021-07-26T06:53:16.408 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509486' 2021-07-26T06:53:16.408 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:16.409 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:53:16.409 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=304942678038 2021-07-26T06:53:16.409 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 304942678038 2021-07-26T06:53:16.409 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509486 3-304942678038' 2021-07-26T06:53:16.410 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:16.410 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:53:16.410 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182440 2021-07-26T06:53:16.410 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182440 2021-07-26T06:53:16.411 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509486 3-304942678038 4-107374182440' 2021-07-26T06:53:16.411 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:16.411 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:53:16.411 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=408021893122 2021-07-26T06:53:16.411 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 408021893122 2021-07-26T06:53:16.412 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509486 3-304942678038 4-107374182440 5-408021893122' 2021-07-26T06:53:16.412 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:16.412 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:53:16.412 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855396 2021-07-26T06:53:16.413 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855396 2021-07-26T06:53:16.413 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509486 3-304942678038 4-107374182440 5-408021893122 6-150323855396' 2021-07-26T06:53:16.413 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:16.413 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:53:16.413 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691875 2021-07-26T06:53:16.414 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691875 2021-07-26T06:53:16.414 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509486 3-304942678038 4-107374182440 5-408021893122 6-150323855396 7-171798691875' 2021-07-26T06:53:16.414 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:17.288 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:53:17.289 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528351 2021-07-26T06:53:17.289 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528351 2021-07-26T06:53:17.289 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509486 3-304942678038 4-107374182440 5-408021893122 6-150323855396 7-171798691875 8-193273528351' 2021-07-26T06:53:17.290 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:17.290 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:53:17.290 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=408021893123 2021-07-26T06:53:17.290 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 408021893123 2021-07-26T06:53:17.290 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509486 3-304942678038 4-107374182440 5-408021893122 6-150323855396 7-171798691875 8-193273528351 9-408021893123' 2021-07-26T06:53:17.291 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:17.291 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836528 2021-07-26T06:53:17.291 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:17.291 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:53:17.292 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836528 2021-07-26T06:53:17.292 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:17.292 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836528 2021-07-26T06:53:17.292 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2021-07-26T06:53:17.292 INFO:tasks.workunit.client.0.smithi071.stderr:279424: waiting osd.0 seq 21474836528 2021-07-26T06:53:17.293 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:53:17.293 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836529 -lt 21474836528 2021-07-26T06:53:17.293 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:17.293 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673007 2021-07-26T06:53:17.293 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:17.294 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:53:17.294 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673007 2021-07-26T06:53:17.294 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:17.294 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673007 2021-07-26T06:53:17.295 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673007' 2021-07-26T06:53:17.295 INFO:tasks.workunit.client.0.smithi071.stderr:279424: waiting osd.1 seq 42949673007 2021-07-26T06:53:17.295 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:53:17.295 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673007 -lt 42949673007 2021-07-26T06:53:17.295 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:17.296 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509486 2021-07-26T06:53:17.296 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:17.296 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:53:17.297 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528352 2021-07-26T06:53:17.297 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528352 2021-07-26T06:53:17.297 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509485 3-304942678037 4-107374182439 5-408021893123 6-150323855395 7-171798691874 8-193273528352' 2021-07-26T06:53:17.297 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:17.298 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:53:17.298 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=408021893122 2021-07-26T06:53:17.298 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 408021893122 2021-07-26T06:53:17.299 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509485 3-304942678037 4-107374182439 5-408021893123 6-150323855395 7-171798691874 8-193273528352 9-408021893122' 2021-07-26T06:53:17.299 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:17.299 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836529 2021-07-26T06:53:17.299 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:17.300 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:53:17.300 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836529 2021-07-26T06:53:17.300 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:17.300 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836529 2021-07-26T06:53:17.300 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2021-07-26T06:53:17.301 INFO:tasks.workunit.client.0.smithi071.stderr:279426: waiting osd.0 seq 21474836529 2021-07-26T06:53:17.301 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:53:17.301 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836529 -lt 21474836529 2021-07-26T06:53:17.301 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:17.301 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673006 2021-07-26T06:53:17.302 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:17.302 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:53:17.302 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673006 2021-07-26T06:53:17.302 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:17.303 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673006 2021-07-26T06:53:17.303 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673006' 2021-07-26T06:53:17.303 INFO:tasks.workunit.client.0.smithi071.stderr:279426: waiting osd.1 seq 42949673006 2021-07-26T06:53:17.303 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:53:17.303 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673007 -lt 42949673006 2021-07-26T06:53:17.304 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:17.304 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509485 2021-07-26T06:53:17.304 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:18.294 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:53:18.294 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509485 2021-07-26T06:53:18.295 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:18.295 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509485 2021-07-26T06:53:18.295 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509485' 2021-07-26T06:53:18.295 INFO:tasks.workunit.client.0.smithi071.stderr:279426: waiting osd.2 seq 64424509485 2021-07-26T06:53:18.296 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:53:18.296 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509486 -lt 64424509485 2021-07-26T06:53:18.296 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:18.296 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-304942678037 2021-07-26T06:53:18.297 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:18.297 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:53:18.297 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-304942678037 2021-07-26T06:53:18.297 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:18.297 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=304942678037 2021-07-26T06:53:18.298 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 304942678037' 2021-07-26T06:53:18.298 INFO:tasks.workunit.client.0.smithi071.stderr:279426: waiting osd.3 seq 304942678037 2021-07-26T06:53:18.298 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:53:18.298 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 304942678038 -lt 304942678037 2021-07-26T06:53:18.299 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:18.299 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182439 2021-07-26T06:53:18.299 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:18.299 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:53:18.300 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182439 2021-07-26T06:53:18.300 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:18.300 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182439 2021-07-26T06:53:18.300 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182439' 2021-07-26T06:53:18.300 INFO:tasks.workunit.client.0.smithi071.stderr:279426: waiting osd.4 seq 107374182439 2021-07-26T06:53:18.301 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:53:18.301 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182441 -lt 107374182439 2021-07-26T06:53:18.301 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:18.301 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-408021893123 2021-07-26T06:53:18.302 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:18.302 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:53:18.302 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-408021893123 2021-07-26T06:53:18.302 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:18.303 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=408021893123 2021-07-26T06:53:18.303 INFO:tasks.workunit.client.0.smithi071.stderr:27e.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:53:18.303 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509486 2021-07-26T06:53:18.304 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:18.304 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509486 2021-07-26T06:53:18.304 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509486' 2021-07-26T06:53:18.304 INFO:tasks.workunit.client.0.smithi071.stderr:279424: waiting osd.2 seq 64424509486 2021-07-26T06:53:18.305 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:53:18.305 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509486 -lt 64424509486 2021-07-26T06:53:18.305 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:18.305 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-304942678038 2021-07-26T06:53:18.306 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:18.306 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:53:18.306 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-304942678038 2021-07-26T06:53:18.306 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:18.306 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=304942678038 2021-07-26T06:53:18.307 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 304942678038' 2021-07-26T06:53:18.307 INFO:tasks.workunit.client.0.smithi071.stderr:279424: waiting osd.3 seq 304942678038 2021-07-26T06:53:18.307 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:53:18.307 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 304942678038 -lt 304942678038 2021-07-26T06:53:18.308 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:18.308 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182440 2021-07-26T06:53:18.308 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:18.308 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:53:18.308 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182440 2021-07-26T06:53:18.309 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:18.309 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182440 2021-07-26T06:53:18.309 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182440' 2021-07-26T06:53:18.309 INFO:tasks.workunit.client.0.smithi071.stderr:279424: waiting osd.4 seq 107374182440 2021-07-26T06:53:18.310 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:53:18.310 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182441 -lt 107374182440 2021-07-26T06:53:18.310 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:18.310 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-408021893122 2021-07-26T06:53:18.311 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:18.311 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:53:18.311 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-408021893122 2021-07-26T06:53:18.311 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:18.311 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=408021893122 2021-07-26T06:53:19.573 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 408021893122' 2021-07-26T06:53:19.574 INFO:tasks.workunit.client.0.smithi071.stderr:279424: waiting osd.5 seq 408021893122 2021-07-26T06:53:19.574 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:53:19.574 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 408021893123 -lt 408021893122 2021-07-26T06:53:19.575 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:19.575 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855396 2021-07-26T06:53:19.575 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:19.575 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:53:19.576 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855396 2021-07-26T06:53:19.576 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:19.576 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855396 2021-07-26T06:53:19.576 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855396' 2021-07-26T06:53:19.576 INFO:tasks.workunit.client.0.smithi071.stderr:279424: waiting osd.6 seq 150323855396 2021-07-26T06:53:19.577 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:53:19.577 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855396 -lt 150323855396 2021-07-26T06:53:19.577 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:19.577 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691875 2021-07-26T06:53:19.578 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:19.578 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:53:19.578 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691875 2021-07-26T06:53:19.578 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:19.578 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691875 2021-07-26T06:53:19.579 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691875' 2021-07-26T06:53:19.579 INFO:tasks.workunit.client.0.smithi071.stderr:279424: waiting osd.7 seq 171798691875 2021-07-26T06:53:19.579 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:53:19.579 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691875 -lt 171798691875 2021-07-26T06:53:19.580 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:19.580 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528351 2021-07-26T06:53:19.580 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:19.580 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:53:19.580 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528351 2021-07-26T06:53:19.581 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:19.581 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528351 2021-07-26T06:53:19.581 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528351' 2021-07-26T06:53:19.581 INFO:tasks.workunit.client.0.smithi071.stderr:279424: waiting osd.8 seq 193273528351 2021-07-26T06:53:19.581 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:53:19.582 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528359426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 408021893123' 2021-07-26T06:53:19.582 INFO:tasks.workunit.client.0.smithi071.stderr:279426: waiting osd.5 seq 408021893123 2021-07-26T06:53:19.583 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:53:19.583 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 408021893123 -lt 408021893123 2021-07-26T06:53:19.583 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:19.583 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855395 2021-07-26T06:53:19.583 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:19.584 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:53:19.584 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855395 2021-07-26T06:53:19.584 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:19.584 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855395 2021-07-26T06:53:19.585 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855395' 2021-07-26T06:53:19.585 INFO:tasks.workunit.client.0.smithi071.stderr:279426: waiting osd.6 seq 150323855395 2021-07-26T06:53:19.585 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:53:19.585 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855396 -lt 150323855395 2021-07-26T06:53:19.585 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:19.586 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691874 2021-07-26T06:53:19.586 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:19.586 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:53:19.586 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691874 2021-07-26T06:53:19.587 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:19.587 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691874 2021-07-26T06:53:19.587 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2021-07-26T06:53:19.587 INFO:tasks.workunit.client.0.smithi071.stderr:279426: waiting osd.7 seq 171798691874 2021-07-26T06:53:19.587 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:53:19.588 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691875 -lt 171798691874 2021-07-26T06:53:19.588 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:19.588 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528352 2021-07-26T06:53:19.588 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:19.589 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:53:19.589 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528352 2021-07-26T06:53:19.589 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:19.589 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528352 2021-07-26T06:53:19.589 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528352' 2021-07-26T06:53:19.590 INFO:tasks.workunit.client.0.smithi071.stderr:279426: waiting osd.8 seq 193273528352 2021-07-26T06:53:19.590 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:53:20.863 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1932735283522 -lt 193273528351 2021-07-26T06:53:20.863 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:20.864 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-408021893123 2021-07-26T06:53:20.864 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:20.864 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:53:20.864 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-408021893123 2021-07-26T06:53:20.865 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:20.865 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=408021893123 2021-07-26T06:53:20.865 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 408021893123' 2021-07-26T06:53:20.865 INFO:tasks.workunit.client.0.smithi071.stderr:279424: waiting osd.9 seq 408021893123 2021-07-26T06:53:20.865 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:53:20.866 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 408021893124 -lt 408021893123 2021-07-26T06:53:20.866 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:53:20.866 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:20.866 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:20.867 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:53:20.867 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:53:20.867 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:53:20.867 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:53:20.868 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:53:20.868 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:53:20.868 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:53:20.868 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:53:20.868 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:53:20.869 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:53:20.869 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:20.869 INFO:tasks.workunit.client.0.smithi071.stderr:279424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:20.869 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:53:20.869 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:53:20.870 INFO:tasks.workunit.client.0.smithi071.stderr:279424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:53:20.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:53:20.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:53:20.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 279423 2021-07-26T06:53:20.905 INFO:tasks.workunit.client.0.smithi071.stderr: -lt 193273528352 2021-07-26T06:53:20.905 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:20.905 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-408021893122 2021-07-26T06:53:20.906 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:20.906 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:53:20.906 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-408021893122 2021-07-26T06:53:20.906 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:20.906 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=408021893122 2021-07-26T06:53:20.907 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 408021893122' 2021-07-26T06:53:20.907 INFO:tasks.workunit.client.0.smithi071.stderr:279426: waiting osd.9 seq 408021893122 2021-07-26T06:53:20.907 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:53:20.907 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 408021893124 -lt 408021893122 2021-07-26T06:53:20.908 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:53:20.908 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:20.908 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:20.908 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:53:20.908 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:53:20.909 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:53:20.909 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:53:20.909 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:53:20.909 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:53:20.909 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:53:20.910 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:53:20.910 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:53:20.910 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:53:20.910 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:20.910 INFO:tasks.workunit.client.0.smithi071.stderr:279426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:20.911 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:53:20.911 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:53:20.911 INFO:tasks.workunit.client.0.smithi071.stderr:279426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:53:20.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:53:20.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:53:20.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:53:20.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:53:20.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2021-07-26T06:53:20.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:53:20.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2021-07-26T06:53:20.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:53:20.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:53:20.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:53:20.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:53:21.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2021-07-26T06:53:21.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2021-07-26T06:53:21.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:53:21.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:53:21.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:53:21.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:53:21.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:53:21.178 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:53:21.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:52:46.610084+0000 2021-07-26T06:53:21.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:53:21.693 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:53:21.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:52:46.610084+0000 2021-07-26T06:53:21.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:53:21.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:52:46.610084+0000 2021-07-26T06:53:21.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:53:21.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:53:21.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:53:21.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:53:21.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:53:21.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:53:21.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:53:21.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:53:21.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:52:46.610084+0000 '>' 2021-07-26T06:52:46.610084+0000 2021-07-26T06:53:21.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:53:22.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:53:22.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:53:22.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:53:22.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:53:22.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:53:22.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:53:22.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:53:23.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:52:46.610084+0000 '>' 2021-07-26T06:52:46.610084+0000 2021-07-26T06:53:23.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:53:24.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:53:24.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:53:24.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:53:24.242 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:53:24.243 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:53:24.243 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:53:24.243 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:53:24.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:52:46.610084+0000 '>' 2021-07-26T06:52:46.610084+0000 2021-07-26T06:53:24.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:53:25.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:53:25.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:53:25.517 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:53:25.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:53:25.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:53:25.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:53:25.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:53:25.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:53:22.463758+0000 '>' 2021-07-26T06:52:46.610084+0000 2021-07-26T06:53:25.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:53:25.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2021-07-26T06:53:25.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:53:25.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:53:25.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:53:25.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 282987"' 2021-07-26T06:53:25.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 282987' 2021-07-26T06:53:25.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:53:25.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:53:25.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:53:25.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 282988"' 2021-07-26T06:53:25.796 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 282988' 2021-07-26T06:53:25.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2021-07-26T06:53:25.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 282987 282988' 2021-07-26T06:53:25.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:53:25.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:53:25.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 282987 2021-07-26T06:53:25.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:53:25.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/282990: /' 2021-07-26T06:53:25.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:53:25.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/282992: /' 2021-07-26T06:53:28.907 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:53:28.908 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:53:28.908 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=5 2021-07-26T06:53:28.908 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:53:28.908 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:53:28.909 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:53:28.909 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:53:28.909 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=5 2021-07-26T06:53:28.909 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:53:28.909 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2021-07-26T06:53:28.910 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:53:28.910 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:53:28.910 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:53:28.910 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:53:28.911 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:53:28.911 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:53:28.911 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:53:28.911 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:53:28.911 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:53:28.912 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=5 2021-07-26T06:53:28.912 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:53:28.912 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:53:28.912 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2021-07-26T06:53:28.913 INFO:tasks.workunit.client.0.smithi071.stderr:282990: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:53:28.913 INFO:tasks.workunit.client.0.smithi071.stderr:282990: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:53:28.913 INFO:tasks.workunit.client.0.smithi071.stderr:282990: _ 2021-07-26T06:53:28.913 INFO:tasks.workunit.client.0.smithi071.stderr:282990: hinfo_key 2021-07-26T06:53:28.913 INFO:tasks.workunit.client.0.smithi071.stderr:282990: snapset 2021-07-26T06:53:28.914 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2021-07-26T06:53:28.914 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:53:28.914 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:53:28.914 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=5 2021-07-26T06:53:28.915 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:53:28.915 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:53:28.915 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:53:28.916 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:53:28.916 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:53:28.916 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:53:28.916 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2021-07-26T06:53:28.917 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2021-07-26T06:53:28.917 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:53:28.918 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:53:28.918 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:53:28.918 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:53:28.918 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:53:28.918 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:53:28.919 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:53:28.919 INFO:tasks.workunit.client.0.smithi071.stderr:282990: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:53:28.919 INFO:tasks.workunit.client.0.smithi071.stderr:282990: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:53:28.919 INFO:tasks.workunit.client.0.smithi071.stderr:282990: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:53:28.919 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:53:28.920 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:53:28.920 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:53:28.920 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:53:28.920 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:53:28.921 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:53:28.921 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:53:28.921 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:53:28.921 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:53:28.922 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:53:28.922 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/5 2021-07-26T06:53:28.922 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.5 2021-07-26T06:53:28.922 INFO:tasks.workunit.client.0.smithi071.stderr:282990: start osd.5 2021-07-26T06:53:29.014 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scru282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:53:29.014 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:53:29.015 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=9 2021-07-26T06:53:29.015 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:53:29.015 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:53:29.015 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:53:29.016 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:53:29.016 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=9 2021-07-26T06:53:29.016 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:53:29.016 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2021-07-26T06:53:29.016 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:53:29.017 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:53:29.017 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:53:29.018 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:53:29.018 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:53:29.018 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:53:29.018 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2021-07-26T06:53:29.018 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:53:29.019 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:53:29.019 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=9 2021-07-26T06:53:29.019 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:53:29.019 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:53:29.020 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2021-07-26T06:53:29.020 INFO:tasks.workunit.client.0.smithi071.stderr:282992: Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:53:29.020 INFO:tasks.workunit.client.0.smithi071.stderr:282992: Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T06:53:29.020 INFO:tasks.workunit.client.0.smithi071.stderr:282992: _ 2021-07-26T06:53:29.021 INFO:tasks.workunit.client.0.smithi071.stderr:282992: hinfo_key 2021-07-26T06:53:29.021 INFO:tasks.workunit.client.0.smithi071.stderr:282992: snapset 2021-07-26T06:53:29.021 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2021-07-26T06:53:29.021 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:53:29.021 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:53:29.022 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=9 2021-07-26T06:53:29.022 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:53:29.022 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/9 2021-07-26T06:53:29.022 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:53:29.023 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:53:29.023 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:53:29.023 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:53:29.024 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2021-07-26T06:53:29.024 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2021-07-26T06:53:29.024 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:53:29.024 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:53:29.024 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:53:29.025 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:53:29.025 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:53:29.025 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:53:29.026 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:53:29.026 INFO:tasks.workunit.client.0.smithi071.stderr:282992: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:53:29.026 INFO:tasks.workunit.client.0.smithi071.stderr:282992: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:53:29.026 INFO:tasks.workunit.client.0.smithi071.stderr:282992: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:53:29.026 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:53:29.027 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:53:29.027 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:53:29.027 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:53:29.027 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:53:29.028 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:53:29.028 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:53:29.028 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:53:29.028 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:53:29.028 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:53:29.029 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/9 2021-07-26T06:53:29.029 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.9 2021-07-26T06:53:29.029 INFO:tasks.workunit.client.0.smithi071.stderr:282992: start osd.9 2021-07-26T06:53:33.116 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 9 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:53:33.117 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/5/whoami 2021-07-26T06:53:33.117 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 5 = 5 ']' 2021-07-26T06:53:33.117 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:53:33.118 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:53:33.118 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:53:33.118 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 2021-07-26T06:53:28.935+0000 7fc7e0d380c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:33.119 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 2021-07-26T06:53:28.937+0000 7fc7e0d380c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:33.119 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 2021-07-26T06:53:28.938+0000 7fc7e0d380c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:33.119 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 5 2021-07-26T06:53:33.119 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:53:33.119 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:53:33.120 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:53:33.120 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:53:33.120 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:33.120 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:53:33.121 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 0 2021-07-26T06:53:33.121 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:33.121 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:53:33.121 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 2021-07-26T06:53:29.458+0000 7fc7e0d380c0 -1 Falling back to public interface 2021-07-26T06:53:33.121 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:33.122 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:33.122 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:33.122 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:53:33.122 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 1 2021-07-26T06:53:33.123 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:33.123 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:53:33.123 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:33.123 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:33.124 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:33.124 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:53:33.124 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 2 2021-07-26T06:53:33.124 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:33.125 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:53:33.125 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:33.125 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 2021-07-26T06:53:32.180+0000 7fc7e0d380c0 -1 osd.5 97 log_to_monitors {default=true} 2021-07-26T06:53:33.125 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:33.125 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:33.126 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:53:33.126 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 3 2021-07-26T06:53:33.126 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:33.417 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cb-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:53:33.417 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/9/whoami 2021-07-26T06:53:33.417 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 9 = 9 ']' 2021-07-26T06:53:33.418 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:53:33.418 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:53:33.418 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:53:33.418 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 2021-07-26T06:53:29.039+0000 7fc845dab0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:33.419 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 2021-07-26T06:53:29.041+0000 7fc845dab0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:33.419 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 2021-07-26T06:53:29.042+0000 7fc845dab0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:33.419 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 9 2021-07-26T06:53:33.420 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:53:33.420 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=9 2021-07-26T06:53:33.420 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:53:33.420 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:53:33.420 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:33.421 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:53:33.421 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 0 2021-07-26T06:53:33.421 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:33.422 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:53:33.422 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 2021-07-26T06:53:29.563+0000 7fc845dab0c0 -1 Falling back to public interface 2021-07-26T06:53:33.422 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:33.422 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:33.423 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:33.423 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:53:33.423 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 1 2021-07-26T06:53:33.423 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:33.423 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:53:33.424 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:33.424 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:33.424 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:33.424 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:53:33.425 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 2 2021-07-26T06:53:33.425 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:33.425 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:53:33.425 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:33.425 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 2021-07-26T06:53:32.500+0000 7fc845dab0c0 -1 osd.9 97 log_to_monitors {default=true} 2021-07-26T06:53:33.426 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:33.426 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:33.426 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:53:33.426 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 3 2021-07-26T06:53:33.427 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:34.491 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:53:34.491 INFO:tasks.workunit.client.0.smithi071.stderr:282990: osd.5 up in weight 1 up_from 101 up_thru 101 down_at 98 last_clean_interval [95,97) [v2:127.0.0.1:6842/283795,v1:127.0.0.1:6843/283795] [v2:127.0.0.1:6844/283795,v1:127.0.0.1:6845/283795] exists,up bbee9dc4-1bba-496c-96c5-7f58e2aee6bc 2021-07-26T06:53:34.491 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:53:34.492 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:53:34.492 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:53:34.492 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:53:34.492 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:53:34.492 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:53:34.493 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:53:34.493 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:53:34.493 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:53:34.494 INFO:tasks.workunit.client.0.smithi071.stderr:282990: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:53:34.495 INFO:tasks.workunit.client.0.smithi071.stderr:282990: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:53:34.495 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:53:34.496 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:53:34.496 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:53:34.496 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:53:34.497 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:53:34.497 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:53:34.498 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:53:34.498 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:53:34.499 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:53:34.499 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 1 2021-07-26T06:53:34.499 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 2 2021-07-26T06:53:34.499 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 3 2021-07-26T06:53:34.500 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 4 2021-07-26T06:53:34.500 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 5 2021-07-26T06:53:34.500 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 6 2021-07-26T06:53:34.500 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 7 2021-07-26T06:53:34.501 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 8 2021-07-26T06:53:34.501 INFO:tasks.workunit.client.0.smithi071.stderr:282990: 9' 2021-07-26T06:53:34.501 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:53:34.501 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:34.501 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:53:34.502 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836534 2021-07-26T06:53:34.502 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836534 2021-07-26T06:53:34.502 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534' 2021-07-26T06:53:34.502 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:34.503 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:53:34.503 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673012 2021-07-26T06:53:34.503 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673012 2021-07-26T06:53:34.503 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534 1-42949673012' 2021-07-26T06:53:34.738 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.clientephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.9 up' 2021-07-26T06:53:34.739 INFO:tasks.workunit.client.0.smithi071.stderr:282992: osd.9 up in weight 1 up_from 102 up_thru 0 down_at 98 last_clean_interval [95,97) [v2:127.0.0.1:6874/283840,v1:127.0.0.1:6875/283840] [v2:127.0.0.1:6876/283840,v1:127.0.0.1:6877/283840] exists,up 1b1672b3-c8a7-4987-b315-589891f1b185 2021-07-26T06:53:34.739 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:53:34.739 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:53:34.739 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:53:34.740 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:53:34.740 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:53:34.740 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:53:34.741 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:53:34.741 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:53:34.742 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:53:34.742 INFO:tasks.workunit.client.0.smithi071.stderr:282992: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:53:34.742 INFO:tasks.workunit.client.0.smithi071.stderr:282992: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:53:34.743 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:53:34.743 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:53:34.743 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:53:34.744 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:53:34.744 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:53:34.745 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:53:34.745 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:53:34.745 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:53:34.745 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:53:34.746 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 1 2021-07-26T06:53:34.746 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 2 2021-07-26T06:53:34.746 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 3 2021-07-26T06:53:34.746 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 4 2021-07-26T06:53:34.747 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 5 2021-07-26T06:53:34.747 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 6 2021-07-26T06:53:34.747 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 7 2021-07-26T06:53:34.747 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 8 2021-07-26T06:53:34.748 INFO:tasks.workunit.client.0.smithi071.stderr:282992: 9' 2021-07-26T06:53:34.748 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:53:34.748 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:34.749 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:53:34.749 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836535 2021-07-26T06:53:34.749 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836535 2021-07-26T06:53:34.749 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836535' 2021-07-26T06:53:34.749 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:34.750 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:53:34.750 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673013 2021-07-26T06:53:34.750 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673013 2021-07-26T06:53:34.751 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836535 1-42949673013' 2021-07-26T06:53:35.529 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:35.529 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:53:35.529 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509491 2021-07-26T06:53:35.530 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509491 2021-07-26T06:53:35.530 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509491' 2021-07-26T06:53:35.530 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:35.530 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:53:35.531 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=304942678043 2021-07-26T06:53:35.531 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 304942678043 2021-07-26T06:53:35.531 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509491 3-304942678043' 2021-07-26T06:53:35.531 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:35.531 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:53:35.532 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182445 2021-07-26T06:53:35.532 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182445 2021-07-26T06:53:35.532 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509491 3-304942678043 4-107374182445' 2021-07-26T06:53:35.532 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:35.533 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:53:35.533 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=433791696898 2021-07-26T06:53:35.533 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 433791696898 2021-07-26T06:53:35.533 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509491 3-304942678043 4-107374182445 5-433791696898' 2021-07-26T06:53:35.533 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:35.534 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:53:35.534 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855401 2021-07-26T06:53:35.534 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855401 2021-07-26T06:53:35.535 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509491 3-304942678043 4-107374182445 5-433791696898 6-150323855401' 2021-07-26T06:53:35.535 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:35.535 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:53:35.535 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691880 2021-07-26T06:53:35.535 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691880 2021-07-26T06:53:35.536 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509491 3-304942678043 4-107374182445 5-433791696898 6-150323855401 7-171798691880' 2021-07-26T06:53:35.536 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:35.654 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:35.654 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:53:35.655 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509492 2021-07-26T06:53:35.655 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509492 2021-07-26T06:53:35.655 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-64424509492' 2021-07-26T06:53:35.655 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:35.656 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:53:35.656 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=304942678044 2021-07-26T06:53:35.656 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 304942678044 2021-07-26T06:53:35.656 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-64424509492 3-304942678044' 2021-07-26T06:53:35.656 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:35.657 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:53:35.657 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182446 2021-07-26T06:53:35.657 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182446 2021-07-26T06:53:35.658 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-64424509492 3-304942678044 4-107374182446' 2021-07-26T06:53:35.658 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:35.658 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:53:35.659 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=433791696899 2021-07-26T06:53:35.659 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 433791696899 2021-07-26T06:53:35.659 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-64424509492 3-304942678044 4-107374182446 5-433791696899' 2021-07-26T06:53:35.659 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:35.660 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:53:35.660 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855402 2021-07-26T06:53:35.660 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855402 2021-07-26T06:53:35.660 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-64424509492 3-304942678044 4-107374182446 5-433791696899 6-150323855402' 2021-07-26T06:53:35.661 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:35.661 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:53:35.661 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691881 2021-07-26T06:53:35.661 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691881 2021-07-26T06:53:35.661 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-64424509492 3-304942678044 4-107374182446 5-433791696899 6-150323855402 7-171798691881' 2021-07-26T06:53:35.662 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:36.411 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/q/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:53:36.412 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528356 2021-07-26T06:53:36.412 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528356 2021-07-26T06:53:36.412 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509491 3-304942678043 4-107374182445 5-433791696898 6-150323855401 7-171798691880 8-193273528356' 2021-07-26T06:53:36.413 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:36.413 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:53:36.413 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=438086664194 2021-07-26T06:53:36.413 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 438086664194 2021-07-26T06:53:36.414 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509491 3-304942678043 4-107374182445 5-433791696898 6-150323855401 7-171798691880 8-193273528356 9-438086664194' 2021-07-26T06:53:36.414 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:36.414 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836534 2021-07-26T06:53:36.414 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:36.416 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:53:36.416 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836534 2021-07-26T06:53:36.417 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:36.417 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836534 2021-07-26T06:53:36.417 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2021-07-26T06:53:36.418 INFO:tasks.workunit.client.0.smithi071.stderr:282990: waiting osd.0 seq 21474836534 2021-07-26T06:53:36.418 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:53:36.418 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836535 -lt 21474836534 2021-07-26T06:53:36.418 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:36.418 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673012 2021-07-26T06:53:36.419 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:36.419 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:53:36.419 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673012 2021-07-26T06:53:36.419 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:36.420 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673012 2021-07-26T06:53:36.420 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673012' 2021-07-26T06:53:36.420 INFO:tasks.workunit.client.0.smithi071.stderr:282990: waiting osd.1 seq 42949673012 2021-07-26T06:53:36.420 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:53:36.420 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673013 -lt 42949673012 2021-07-26T06:53:36.421 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:36.421 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509491 2021-07-26T06:53:36.421 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:36.587 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/ca/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:53:36.587 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528358 2021-07-26T06:53:36.587 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528358 2021-07-26T06:53:36.588 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-64424509492 3-304942678044 4-107374182446 5-433791696899 6-150323855402 7-171798691881 8-193273528358' 2021-07-26T06:53:36.588 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:36.588 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:53:36.588 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=438086664195 2021-07-26T06:53:36.589 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 438086664195 2021-07-26T06:53:36.589 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-64424509492 3-304942678044 4-107374182446 5-433791696899 6-150323855402 7-171798691881 8-193273528358 9-438086664195' 2021-07-26T06:53:36.589 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:36.589 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836535 2021-07-26T06:53:36.589 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:36.590 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:53:36.590 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836535 2021-07-26T06:53:36.590 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:36.590 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836535 2021-07-26T06:53:36.591 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2021-07-26T06:53:36.591 INFO:tasks.workunit.client.0.smithi071.stderr:282992: waiting osd.0 seq 21474836535 2021-07-26T06:53:36.591 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:53:36.591 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836535 -lt 21474836535 2021-07-26T06:53:36.592 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:36.592 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673013 2021-07-26T06:53:36.592 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:36.592 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:53:36.593 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673013 2021-07-26T06:53:36.593 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:36.593 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673013 2021-07-26T06:53:36.593 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673013' 2021-07-26T06:53:36.593 INFO:tasks.workunit.client.0.smithi071.stderr:282992: waiting osd.1 seq 42949673013 2021-07-26T06:53:36.594 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:53:36.594 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673013 -lt 42949673013 2021-07-26T06:53:36.594 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:36.594 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509492 2021-07-26T06:53:36.594 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:37.338 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cepephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:53:37.338 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509491 2021-07-26T06:53:37.338 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:37.339 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509491 2021-07-26T06:53:37.339 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509491' 2021-07-26T06:53:37.339 INFO:tasks.workunit.client.0.smithi071.stderr:282990: waiting osd.2 seq 64424509491 2021-07-26T06:53:37.339 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:53:37.340 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509492 -lt 64424509491 2021-07-26T06:53:37.340 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:37.340 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-304942678043 2021-07-26T06:53:37.341 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:37.341 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:53:37.341 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-304942678043 2021-07-26T06:53:37.341 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:37.342 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=304942678043 2021-07-26T06:53:37.342 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 304942678043' 2021-07-26T06:53:37.342 INFO:tasks.workunit.client.0.smithi071.stderr:282990: waiting osd.3 seq 304942678043 2021-07-26T06:53:37.342 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:53:37.342 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 304942678044 -lt 304942678043 2021-07-26T06:53:37.343 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:37.343 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182445 2021-07-26T06:53:37.343 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:37.343 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:53:37.344 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182445 2021-07-26T06:53:37.344 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:37.344 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182445 2021-07-26T06:53:37.344 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182445' 2021-07-26T06:53:37.344 INFO:tasks.workunit.client.0.smithi071.stderr:282990: waiting osd.4 seq 107374182445 2021-07-26T06:53:37.345 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:53:37.345 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182446 -lt 107374182445 2021-07-26T06:53:37.345 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:37.345 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-433791696898 2021-07-26T06:53:37.345 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:37.346 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:53:37.346 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-433791696898 2021-07-26T06:53:37.346 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:37.466 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=43htest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:53:37.466 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509492 2021-07-26T06:53:37.467 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:37.467 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509492 2021-07-26T06:53:37.468 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509492' 2021-07-26T06:53:37.468 INFO:tasks.workunit.client.0.smithi071.stderr:282992: waiting osd.2 seq 64424509492 2021-07-26T06:53:37.469 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:53:37.469 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509492 -lt 64424509492 2021-07-26T06:53:37.469 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:37.470 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-304942678044 2021-07-26T06:53:37.471 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:37.471 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:53:37.471 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-304942678044 2021-07-26T06:53:37.472 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:37.472 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=304942678044 2021-07-26T06:53:37.473 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 304942678044' 2021-07-26T06:53:37.473 INFO:tasks.workunit.client.0.smithi071.stderr:282992: waiting osd.3 seq 304942678044 2021-07-26T06:53:37.473 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:53:37.474 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 304942678044 -lt 304942678044 2021-07-26T06:53:37.474 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:37.475 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182446 2021-07-26T06:53:37.475 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:37.476 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:53:37.476 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182446 2021-07-26T06:53:37.477 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:37.477 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182446 2021-07-26T06:53:37.477 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2021-07-26T06:53:37.478 INFO:tasks.workunit.client.0.smithi071.stderr:282992: waiting osd.4 seq 107374182446 2021-07-26T06:53:37.478 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:53:37.478 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182446 -lt 107374182446 2021-07-26T06:53:37.479 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:37.479 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-433791696899 2021-07-26T06:53:37.479 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:37.479 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:53:37.480 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-433791696899 2021-07-26T06:53:37.480 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:38.528 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=43373791696898 2021-07-26T06:53:38.528 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 433791696898' 2021-07-26T06:53:38.528 INFO:tasks.workunit.client.0.smithi071.stderr:282990: waiting osd.5 seq 433791696898 2021-07-26T06:53:38.529 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:53:38.529 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 433791696899 -lt 433791696898 2021-07-26T06:53:38.529 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:38.529 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855401 2021-07-26T06:53:38.529 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:38.530 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:53:38.530 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855401 2021-07-26T06:53:38.530 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:38.530 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855401 2021-07-26T06:53:38.531 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855401' 2021-07-26T06:53:38.531 INFO:tasks.workunit.client.0.smithi071.stderr:282990: waiting osd.6 seq 150323855401 2021-07-26T06:53:38.531 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:53:38.531 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855402 -lt 150323855401 2021-07-26T06:53:38.531 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:38.532 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691880 2021-07-26T06:53:38.532 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:38.532 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:53:38.533 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691880 2021-07-26T06:53:38.533 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:38.533 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691880 2021-07-26T06:53:38.533 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691880' 2021-07-26T06:53:38.534 INFO:tasks.workunit.client.0.smithi071.stderr:282990: waiting osd.7 seq 171798691880 2021-07-26T06:53:38.534 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:53:38.534 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691881 -lt 171798691880 2021-07-26T06:53:38.534 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:38.534 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528356 2021-07-26T06:53:38.535 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:38.535 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:53:38.535 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528356 2021-07-26T06:53:38.535 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:38.536 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528356 2021-07-26T06:53:38.536 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528356' 2021-07-26T06:53:38.536 INFO:tasks.workunit.client.0.smithi071.stderr:282990: waiting osd.8 seq 193273528356 2021-07-26T06:53:38.536 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:53:38.677 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test91696899 2021-07-26T06:53:38.677 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 433791696899' 2021-07-26T06:53:38.677 INFO:tasks.workunit.client.0.smithi071.stderr:282992: waiting osd.5 seq 433791696899 2021-07-26T06:53:38.678 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:53:38.678 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 433791696899 -lt 433791696899 2021-07-26T06:53:38.678 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:38.678 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855402 2021-07-26T06:53:38.679 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:38.679 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:53:38.679 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855402 2021-07-26T06:53:38.679 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:38.680 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855402 2021-07-26T06:53:38.680 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855402' 2021-07-26T06:53:38.680 INFO:tasks.workunit.client.0.smithi071.stderr:282992: waiting osd.6 seq 150323855402 2021-07-26T06:53:38.680 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:53:38.680 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855402 -lt 150323855402 2021-07-26T06:53:38.681 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:38.681 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691881 2021-07-26T06:53:38.681 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:38.681 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:53:38.682 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691881 2021-07-26T06:53:38.682 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:38.682 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691881 2021-07-26T06:53:38.682 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691881' 2021-07-26T06:53:38.683 INFO:tasks.workunit.client.0.smithi071.stderr:282992: waiting osd.7 seq 171798691881 2021-07-26T06:53:38.683 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:53:38.683 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691881 -lt 171798691881 2021-07-26T06:53:38.683 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:38.683 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528358 2021-07-26T06:53:38.684 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:38.684 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:53:38.684 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528358 2021-07-26T06:53:38.684 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:38.684 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528358 2021-07-26T06:53:38.685 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2021-07-26T06:53:38.685 INFO:tasks.workunit.client.0.smithi071.stderr:282992: waiting osd.8 seq 193273528358 2021-07-26T06:53:38.685 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:53:40.145 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 193273528358 -lt 193273528356 2021-07-26T06:53:40.145 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:40.145 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-438086664194 2021-07-26T06:53:40.146 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:40.146 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:53:40.146 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-438086664194 2021-07-26T06:53:40.146 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:40.147 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=438086664194 2021-07-26T06:53:40.147 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 438086664194' 2021-07-26T06:53:40.147 INFO:tasks.workunit.client.0.smithi071.stderr:282990: waiting osd.9 seq 438086664194 2021-07-26T06:53:40.147 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:53:40.148 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 438086664195 -lt 438086664194 2021-07-26T06:53:40.148 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:53:40.148 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:40.148 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:40.149 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:53:40.149 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:53:40.149 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:53:40.149 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:53:40.150 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:53:40.150 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:53:40.150 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:53:40.150 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:53:40.150 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:53:40.151 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:53:40.151 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:40.151 INFO:tasks.workunit.client.0.smithi071.stderr:282990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:40.151 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:53:40.151 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:53:40.152 INFO:tasks.workunit.client.0.smithi071.stderr:282990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:53:40.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:53:40.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:53:40.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 282988 2021-07-26T06:53:40.154 INFO:tasks.workunit.client.0.smithi071.stderr:93273528358 -lt 193273528358 2021-07-26T06:53:40.154 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:40.154 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-438086664195 2021-07-26T06:53:40.154 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:40.155 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:53:40.155 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-438086664195 2021-07-26T06:53:40.155 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:40.155 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=438086664195 2021-07-26T06:53:40.156 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 438086664195' 2021-07-26T06:53:40.156 INFO:tasks.workunit.client.0.smithi071.stderr:282992: waiting osd.9 seq 438086664195 2021-07-26T06:53:40.156 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:53:40.156 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 438086664195 -lt 438086664195 2021-07-26T06:53:40.156 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:53:40.157 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:40.157 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:40.157 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:53:40.157 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:53:40.157 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:53:40.158 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:53:40.158 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:53:40.158 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:53:40.158 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:53:40.159 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:53:40.159 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:53:40.159 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:53:40.159 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:40.160 INFO:tasks.workunit.client.0.smithi071.stderr:282992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:40.160 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:53:40.160 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:53:40.160 INFO:tasks.workunit.client.0.smithi071.stderr:282992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:53:40.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:53:40.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:53:40.161 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:53:40.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:53:40.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2021-07-26T06:53:40.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:53:40.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:53:40.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:53:40.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2021-07-26T06:53:40.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2021-07-26T06:53:40.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2021-07-26T06:53:40.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=3 2021-07-26T06:53:40.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=5 2021-07-26T06:53:40.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2021-07-26T06:53:40.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:53:40.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:53:40.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:53:40.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 286390"' 2021-07-26T06:53:40.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 286390' 2021-07-26T06:53:40.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:53:40.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:53:40.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:53:40.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 286391"' 2021-07-26T06:53:40.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 286391' 2021-07-26T06:53:40.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:53:40.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2021-07-26T06:53:40.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 286390 286391' 2021-07-26T06:53:40.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:53:40.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:53:40.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 286390 2021-07-26T06:53:40.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/286393: /' 2021-07-26T06:53:40.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:53:40.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/286395: /' 2021-07-26T06:53:43.212 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:53:43.212 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:53:43.212 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:53:43.213 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:53:43.213 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:53:43.213 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:53:43.213 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:53:43.213 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:53:43.214 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:53:43.214 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:53:43.214 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:53:43.214 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:53:43.215 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:53:43.215 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:53:43.215 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:53:43.215 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:53:43.216 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T06:53:43.216 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:53:43.216 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:53:43.216 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:53:43.217 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:53:43.217 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:53:43.217 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2021-07-26T06:53:43.217 INFO:tasks.workunit.client.0.smithi071.stderr:286393: Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:53:43.217 INFO:tasks.workunit.client.0.smithi071.stderr:286393: remove 0#2:eb822e21:::SOMETHING:head# 2021-07-26T06:53:43.218 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:53:43.218 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:53:43.218 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:53:43.218 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:53:43.219 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:53:43.219 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:53:43.219 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:53:43.220 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:53:43.220 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:53:43.220 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:53:43.221 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:53:43.221 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:53:43.221 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:53:43.221 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:53:43.222 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:53:43.222 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:53:43.222 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:53:43.222 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:53:43.223 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:53:43.223 INFO:tasks.workunit.client.0.smithi071.stderr:286393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:53:43.223 INFO:tasks.workunit.client.0.smithi071.stderr:286393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:53:43.223 INFO:tasks.workunit.client.0.smithi071.stderr:286393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:53:43.223 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:53:43.224 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:53:43.224 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:53:43.224 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:53:43.224 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:53:43.225 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:53:43.225 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:53:43.225 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:53:43.225 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:53:43.225 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:53:43.226 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:53:43.226 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:53:43.226 INFO:tasks.workunit.client.0.smithi071.stderr:286393: start osd.3 2021-07-26T06:53:43.276 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-ex286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:53:43.276 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:53:43.276 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=5 2021-07-26T06:53:43.277 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:53:43.277 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:53:43.277 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:53:43.277 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:53:43.278 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=5 2021-07-26T06:53:43.278 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:53:43.278 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2021-07-26T06:53:43.278 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:53:43.279 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:53:43.279 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:53:43.279 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:53:43.279 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:53:43.280 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:53:43.280 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2021-07-26T06:53:43.280 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:53:43.280 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:53:43.281 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=5 2021-07-26T06:53:43.281 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:53:43.281 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:53:43.281 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2021-07-26T06:53:43.281 INFO:tasks.workunit.client.0.smithi071.stderr:286395: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:53:43.282 INFO:tasks.workunit.client.0.smithi071.stderr:286395: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:53:43.282 INFO:tasks.workunit.client.0.smithi071.stderr:286395: remove 1#2:eb822e21:::SOMETHING:head# 2021-07-26T06:53:43.282 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2021-07-26T06:53:43.283 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:53:43.283 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:53:43.283 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=5 2021-07-26T06:53:43.283 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:53:43.284 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:53:43.284 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:53:43.284 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:53:43.285 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:53:43.285 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:53:43.285 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2021-07-26T06:53:43.285 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2021-07-26T06:53:43.286 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:53:43.286 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:53:43.286 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:53:43.286 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:53:43.287 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:53:43.287 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:53:43.287 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:53:43.287 INFO:tasks.workunit.client.0.smithi071.stderr:286395: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:53:43.288 INFO:tasks.workunit.client.0.smithi071.stderr:286395: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:53:43.288 INFO:tasks.workunit.client.0.smithi071.stderr:286395: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:53:43.288 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:53:43.288 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:53:43.288 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:53:43.289 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:53:43.289 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:53:43.289 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:53:43.289 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:53:43.290 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:53:43.290 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:53:43.290 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:53:43.290 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/5 2021-07-26T06:53:43.291 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.5 2021-07-26T06:53:43.291 INFO:tasks.workunit.client.0.smithi071.stderr:286395: start osd.5 2021-07-26T06:53:47.571 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:53:47.572 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/5/whoami 2021-07-26T06:53:47.572 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 5 = 5 ']' 2021-07-26T06:53:47.572 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:53:47.573 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:53:47.573 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:53:47.573 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 2021-07-26T06:53:43.301+0000 7fd64ff320c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:47.573 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 2021-07-26T06:53:43.303+0000 7fd64ff320c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:47.573 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 2021-07-26T06:53:43.304+0000 7fd64ff320c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:47.574 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 5 2021-07-26T06:53:47.574 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:53:47.574 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:53:47.574 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:53:47.575 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:53:47.575 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:47.575 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:53:47.575 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 0 2021-07-26T06:53:47.576 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:47.576 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:53:47.576 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 2021-07-26T06:53:43.820+0000 7fd64ff320c0 -1 Falling back to public interface 2021-07-26T06:53:47.576 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:47.576 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:47.577 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:47.577 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:53:47.577 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 1 2021-07-26T06:53:47.577 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:47.578 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:53:47.578 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:47.578 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:47.579 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:47.579 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:53:47.580 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 2 2021-07-26T06:53:47.580 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:47.580 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:53:47.581 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:47.581 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 2021-07-26T06:53:46.772+0000 7fd64ff320c0 -1 osd.5 104 log_to_monitors {default=true} 2021-07-26T06:53:47.581 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:47.582 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:47.582 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:53:47.583 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 3 2021-07-26T06:53:47.583 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:47.894 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/cperimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:53:47.895 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:53:47.895 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:53:47.896 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:53:47.896 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:53:47.897 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:53:47.897 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 2021-07-26T06:53:43.239+0000 7f4585f2c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:47.898 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 2021-07-26T06:53:43.241+0000 7f4585f2c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:47.898 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 2021-07-26T06:53:43.242+0000 7f4585f2c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:53:47.898 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:53:47.899 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:53:47.899 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:53:47.900 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:53:47.900 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:53:47.901 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:47.901 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:53:47.902 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 0 2021-07-26T06:53:47.902 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:47.903 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:53:47.903 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 2021-07-26T06:53:43.760+0000 7f4585f2c0c0 -1 Falling back to public interface 2021-07-26T06:53:47.904 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:47.904 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:47.905 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:47.905 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:53:47.905 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 1 2021-07-26T06:53:47.906 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:47.906 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:53:47.907 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:47.907 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:47.908 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:47.909 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:53:47.909 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 2 2021-07-26T06:53:47.909 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:47.910 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:53:47.910 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 2021-07-26T06:53:46.554+0000 7f4585f2c0c0 -1 osd.3 104 log_to_monitors {default=true} 2021-07-26T06:53:47.911 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:53:47.911 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:53:47.912 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:53:47.912 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:53:47.913 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 3 2021-07-26T06:53:47.913 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:53:47.913 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:53:48.650 INFO:tasks.workunit.client.0.smithi071.stderr:286393: osd.3 up in weight 1 up_from 109 up_thru 102 down_at 105 last_clean_interval [71,104) [v2:127.0.0.1:6826/287198,v1:127.0.0.1:6827/287198] [v2:127.0.0.1:6828/287198,v1:127.0.0.1:6829/287198] exists,up 9c55da62-886d-4a69-a717-2a90c8e238a9 2021-07-26T06:53:48.650 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:53:48.650 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:53:48.651 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:53:48.651 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:53:48.651 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:53:48.651 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:53:48.652 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:53:48.652 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:53:48.652 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:53:48.652 INFO:tasks.workunit.client.0.smithi071.stderr:286393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:53:48.653 INFO:tasks.workunit.client.0.smithi071.stderr:286393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:53:48.653 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:53:48.653 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:53:48.653 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:53:48.653 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:53:48.654 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:53:48.654 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:53:48.654 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:53:48.655 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:53:48.655 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:53:48.655 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 1 2021-07-26T06:53:48.655 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 2 2021-07-26T06:53:48.656 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 3 2021-07-26T06:53:48.656 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 4 2021-07-26T06:53:48.656 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 5 2021-07-26T06:53:48.656 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 6 2021-07-26T06:53:48.657 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 7 2021-07-26T06:53:48.657 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 8 2021-07-26T06:53:48.657 INFO:tasks.workunit.client.0.smithi071.stderr:286393: 9' 2021-07-26T06:53:48.657 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:53:48.657 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:48.658 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:53:48.658 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836539 2021-07-26T06:53:48.658 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836539 2021-07-26T06:53:48.658 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836539' 2021-07-26T06:53:48.659 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:48.659 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:53:48.659 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673017 2021-07-26T06:53:48.659 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673017 2021-07-26T06:53:48.659 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836539 1-42949673017' 2021-07-26T06:53:48.660 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:48.660 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntulone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:53:48.660 INFO:tasks.workunit.client.0.smithi071.stderr:286395: osd.5 up in weight 1 up_from 109 up_thru 101 down_at 105 last_clean_interval [101,104) [v2:127.0.0.1:6842/287224,v1:127.0.0.1:6843/287224] [v2:127.0.0.1:6844/287224,v1:127.0.0.1:6845/287224] exists,up bbee9dc4-1bba-496c-96c5-7f58e2aee6bc 2021-07-26T06:53:48.660 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:53:48.661 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:53:48.661 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:53:48.661 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:53:48.661 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:53:48.661 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:53:48.662 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:53:48.662 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:53:48.662 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:53:48.662 INFO:tasks.workunit.client.0.smithi071.stderr:286395: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:53:48.662 INFO:tasks.workunit.client.0.smithi071.stderr:286395: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:53:48.663 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:53:48.663 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:53:48.663 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:53:48.663 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:53:48.664 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:53:48.664 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:53:48.664 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:53:48.664 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:53:48.664 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:53:48.665 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 1 2021-07-26T06:53:48.665 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 2 2021-07-26T06:53:48.666 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 3 2021-07-26T06:53:48.666 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 4 2021-07-26T06:53:48.666 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 5 2021-07-26T06:53:48.666 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 6 2021-07-26T06:53:48.666 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 7 2021-07-26T06:53:48.667 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 8 2021-07-26T06:53:48.667 INFO:tasks.workunit.client.0.smithi071.stderr:286395: 9' 2021-07-26T06:53:48.667 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:53:48.667 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:48.668 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:53:48.668 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836540 2021-07-26T06:53:48.668 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836540 2021-07-26T06:53:48.668 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836540' 2021-07-26T06:53:48.668 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:48.669 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:53:48.669 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673018 2021-07-26T06:53:48.669 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673018 2021-07-26T06:53:48.669 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836540 1-42949673018' 2021-07-26T06:53:49.433 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:49.433 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:53:49.433 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509497 2021-07-26T06:53:49.434 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509497 2021-07-26T06:53:49.434 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836540 1-42949673018 2-64424509497' 2021-07-26T06:53:49.434 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:49.434 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:53:49.435 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=468151435267 2021-07-26T06:53:49.435 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 468151435267 2021-07-26T06:53:49.435 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836540 1-42949673018 2-64424509497 3-468151435267' 2021-07-26T06:53:49.435 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:49.435 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:53:49.436 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182451 2021-07-26T06:53:49.436 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182451 2021-07-26T06:53:49.436 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836540 1-42949673018 2-64424509497 3-468151435267 4-107374182451' 2021-07-26T06:53:49.436 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:49.436 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:53:49.437 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=468151435266 2021-07-26T06:53:49.437 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 468151435266 2021-07-26T06:53:49.437 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836540 1-42949673018 2-64424509497 3-468151435267 4-107374182451 5-468151435266' 2021-07-26T06:53:49.437 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:49.438 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:53:49.438 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855407 2021-07-26T06:53:49.438 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855407 2021-07-26T06:53:49.438 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836540 1-42949673018 2-64424509497 3-468151435267 4-107374182451 5-468151435266 6-150323855407' 2021-07-26T06:53:49.438 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:49.439 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:53:49.439 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691885 2021-07-26T06:53:49.439 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691885 2021-07-26T06:53:49.439 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836540 1-42949673018 2-64424509497 3-468151435267 4-107374182451 5-468151435266 6-150323855407 7-171798691885' 2021-07-26T06:53:49.440 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:49.557 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/st/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:53:49.557 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509496 2021-07-26T06:53:49.558 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509496 2021-07-26T06:53:49.558 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-64424509496' 2021-07-26T06:53:49.558 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:49.558 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:53:49.558 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=468151435266 2021-07-26T06:53:49.559 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 468151435266 2021-07-26T06:53:49.559 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-64424509496 3-468151435266' 2021-07-26T06:53:49.559 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:49.559 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:53:49.560 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182450 2021-07-26T06:53:49.560 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182450 2021-07-26T06:53:49.560 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-64424509496 3-468151435266 4-107374182450' 2021-07-26T06:53:49.561 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:49.561 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:53:49.561 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=468151435267 2021-07-26T06:53:49.561 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 468151435267 2021-07-26T06:53:49.561 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-64424509496 3-468151435266 4-107374182450 5-468151435267' 2021-07-26T06:53:49.562 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:49.562 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:53:49.562 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855406 2021-07-26T06:53:49.562 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855406 2021-07-26T06:53:49.563 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-64424509496 3-468151435266 4-107374182450 5-468151435267 6-150323855406' 2021-07-26T06:53:49.563 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:49.563 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:53:49.563 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691884 2021-07-26T06:53:49.564 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691884 2021-07-26T06:53:49.564 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-64424509496 3-468151435266 4-107374182450 5-468151435267 6-150323855406 7-171798691884' 2021-07-26T06:53:49.564 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:49.564 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:53:50.262 INFO:tasks.workunit.client.0.smithi071.stderr:286393: andalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:53:50.262 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528361 2021-07-26T06:53:50.263 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528361 2021-07-26T06:53:50.263 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836540 1-42949673018 2-64424509497 3-468151435267 4-107374182451 5-468151435266 6-150323855407 7-171798691885 8-193273528361' 2021-07-26T06:53:50.264 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:50.264 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:53:50.265 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=438086664199 2021-07-26T06:53:50.265 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 438086664199 2021-07-26T06:53:50.265 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836540 1-42949673018 2-64424509497 3-468151435267 4-107374182451 5-468151435266 6-150323855407 7-171798691885 8-193273528361 9-438086664199' 2021-07-26T06:53:50.265 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:50.266 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836540 2021-07-26T06:53:50.266 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:50.267 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:53:50.268 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836540 2021-07-26T06:53:50.268 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:50.268 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836540 2021-07-26T06:53:50.269 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2021-07-26T06:53:50.269 INFO:tasks.workunit.client.0.smithi071.stderr:286395: waiting osd.0 seq 21474836540 2021-07-26T06:53:50.269 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:53:50.269 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836540 -lt 21474836540 2021-07-26T06:53:50.270 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:50.270 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673018 2021-07-26T06:53:50.270 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:50.270 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:53:50.271 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673018 2021-07-26T06:53:50.271 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:50.271 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673018 2021-07-26T06:53:50.272 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673018' 2021-07-26T06:53:50.272 INFO:tasks.workunit.client.0.smithi071.stderr:286395: waiting osd.1 seq 42949673018 2021-07-26T06:53:50.272 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:53:50.272 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673018 -lt 42949673018 2021-07-26T06:53:50.272 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:50.273 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509497 2021-07-26T06:53:50.273 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:50.273 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtes/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528362 2021-07-26T06:53:50.274 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528362 2021-07-26T06:53:50.274 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-64424509496 3-468151435266 4-107374182450 5-468151435267 6-150323855406 7-171798691884 8-193273528362' 2021-07-26T06:53:50.274 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:53:50.274 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:53:50.275 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=438086664200 2021-07-26T06:53:50.275 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 438086664200 2021-07-26T06:53:50.275 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-64424509496 3-468151435266 4-107374182450 5-468151435267 6-150323855406 7-171798691884 8-193273528362 9-438086664200' 2021-07-26T06:53:50.275 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:50.276 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836539 2021-07-26T06:53:50.276 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:50.276 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:53:50.276 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836539 2021-07-26T06:53:50.276 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:50.277 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836539 2021-07-26T06:53:50.277 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2021-07-26T06:53:50.277 INFO:tasks.workunit.client.0.smithi071.stderr:286393: waiting osd.0 seq 21474836539 2021-07-26T06:53:50.278 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:53:50.278 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836540 -lt 21474836539 2021-07-26T06:53:50.278 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:50.278 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673017 2021-07-26T06:53:50.279 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:50.279 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:53:50.279 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673017 2021-07-26T06:53:50.279 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:50.280 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673017 2021-07-26T06:53:50.280 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673017' 2021-07-26T06:53:50.280 INFO:tasks.workunit.client.0.smithi071.stderr:286393: waiting osd.1 seq 42949673017 2021-07-26T06:53:50.280 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:53:50.281 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673018 -lt 42949673017 2021-07-26T06:53:50.281 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:50.281 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509496 2021-07-26T06:53:50.282 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:50.282 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:53:51.170 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509496 2021-07-26T06:53:51.170 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:51.171 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509496 2021-07-26T06:53:51.171 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2021-07-26T06:53:51.171 INFO:tasks.workunit.client.0.smithi071.stderr:286393: waiting osd.2 seq 64424509496 2021-07-26T06:53:51.172 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:53:51.172 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509497 -lt 64424509496 2021-07-26T06:53:51.172 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:51.172 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-468151435266 2021-07-26T06:53:51.173 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:51.173 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:53:51.173 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-468151435266 2021-07-26T06:53:51.173 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:51.173 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=468151435266 2021-07-26T06:53:51.174 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 468151435266' 2021-07-26T06:53:51.174 INFO:tasks.workunit.client.0.smithi071.stderr:286393: waiting osd.3 seq 468151435266 2021-07-26T06:53:51.174 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:53:51.174 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 468151435267 -lt 468151435266 2021-07-26T06:53:51.175 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:51.175 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182450 2021-07-26T06:53:51.175 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:51.175 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:53:51.175 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182450 2021-07-26T06:53:51.176 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:51.176 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182450 2021-07-26T06:53:51.176 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182450' 2021-07-26T06:53:51.176 INFO:tasks.workunit.client.0.smithi071.stderr:286393: waiting osd.4 seq 107374182450 2021-07-26T06:53:51.177 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:53:51.177 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182451 -lt 107374182450 2021-07-26T06:53:51.177 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:51.177 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-468151435267 2021-07-26T06:53:51.177 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:51.178 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:53:51.178 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-468151435267 2021-07-26T06:53:51.178 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:51.178 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=468151435267 2021-07-26T06:53:51.179 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: ft/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:53:51.179 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509497 2021-07-26T06:53:51.179 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:51.180 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509497 2021-07-26T06:53:51.180 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509497' 2021-07-26T06:53:51.180 INFO:tasks.workunit.client.0.smithi071.stderr:286395: waiting osd.2 seq 64424509497 2021-07-26T06:53:51.180 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:53:51.180 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509497 -lt 64424509497 2021-07-26T06:53:51.181 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:51.181 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-468151435267 2021-07-26T06:53:51.181 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:51.181 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:53:51.182 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-468151435267 2021-07-26T06:53:51.182 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:51.182 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=468151435267 2021-07-26T06:53:51.182 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 468151435267' 2021-07-26T06:53:51.183 INFO:tasks.workunit.client.0.smithi071.stderr:286395: waiting osd.3 seq 468151435267 2021-07-26T06:53:51.183 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:53:51.183 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 468151435267 -lt 468151435267 2021-07-26T06:53:51.183 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:51.183 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182451 2021-07-26T06:53:51.184 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:51.184 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:53:51.184 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182451 2021-07-26T06:53:51.184 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:51.185 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182451 2021-07-26T06:53:51.185 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2021-07-26T06:53:51.185 INFO:tasks.workunit.client.0.smithi071.stderr:286395: waiting osd.4 seq 107374182451 2021-07-26T06:53:51.185 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:53:51.185 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182451 -lt 107374182451 2021-07-26T06:53:51.186 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:51.186 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-468151435266 2021-07-26T06:53:51.186 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:51.186 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:53:51.186 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-468151435266 2021-07-26T06:53:51.187 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:52.436 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=46815143lush_pg_stats: echo 'waiting osd.5 seq 468151435267' 2021-07-26T06:53:52.437 INFO:tasks.workunit.client.0.smithi071.stderr:286393: waiting osd.5 seq 468151435267 2021-07-26T06:53:52.437 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:53:52.437 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 468151435267 -lt 468151435267 2021-07-26T06:53:52.437 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:52.438 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855406 2021-07-26T06:53:52.438 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:52.438 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:53:52.438 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855406 2021-07-26T06:53:52.439 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:52.439 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855406 2021-07-26T06:53:52.439 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855406' 2021-07-26T06:53:52.439 INFO:tasks.workunit.client.0.smithi071.stderr:286393: waiting osd.6 seq 150323855406 2021-07-26T06:53:52.439 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:53:52.440 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855407 -lt 150323855406 2021-07-26T06:53:52.440 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:52.441 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691884 2021-07-26T06:53:52.441 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:52.441 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:53:52.441 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691884 2021-07-26T06:53:52.442 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:52.442 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691884 2021-07-26T06:53:52.442 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2021-07-26T06:53:52.442 INFO:tasks.workunit.client.0.smithi071.stderr:286393: waiting osd.7 seq 171798691884 2021-07-26T06:53:52.442 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:53:52.443 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691886 -lt 171798691884 2021-07-26T06:53:52.443 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:52.443 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528362 2021-07-26T06:53:52.443 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:52.444 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:53:52.444 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528362 2021-07-26T06:53:52.444 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:52.444 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528362 2021-07-26T06:53:52.444 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528362' 2021-07-26T06:53:52.445 INFO:tasks.workunit.client.0.smithi071.stderr:286393: waiting osd.8 seq 193273528362 2021-07-26T06:53:52.445 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:53:52.445 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528363 -lt 193273528362 2021-07-26T06:53:52.446 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cep5266 2021-07-26T06:53:52.446 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 468151435266' 2021-07-26T06:53:52.446 INFO:tasks.workunit.client.0.smithi071.stderr:286395: waiting osd.5 seq 468151435266 2021-07-26T06:53:52.447 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:53:52.447 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 468151435267 -lt 468151435266 2021-07-26T06:53:52.447 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:52.447 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855407 2021-07-26T06:53:52.447 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:52.448 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:53:52.448 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855407 2021-07-26T06:53:52.448 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:52.448 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855407 2021-07-26T06:53:52.449 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855407' 2021-07-26T06:53:52.449 INFO:tasks.workunit.client.0.smithi071.stderr:286395: waiting osd.6 seq 150323855407 2021-07-26T06:53:52.449 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:53:52.449 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855407 -lt 150323855407 2021-07-26T06:53:52.449 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:52.450 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691885 2021-07-26T06:53:52.450 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:52.450 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:53:52.450 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691885 2021-07-26T06:53:52.451 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:52.451 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691885 2021-07-26T06:53:52.451 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691885' 2021-07-26T06:53:52.451 INFO:tasks.workunit.client.0.smithi071.stderr:286395: waiting osd.7 seq 171798691885 2021-07-26T06:53:52.452 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:53:52.452 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691886 -lt 171798691885 2021-07-26T06:53:52.452 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:52.452 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528361 2021-07-26T06:53:52.452 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:52.453 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:53:52.453 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528361 2021-07-26T06:53:52.453 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:52.453 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528361 2021-07-26T06:53:52.454 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2021-07-26T06:53:52.454 INFO:tasks.workunit.client.0.smithi071.stderr:286395: waiting osd.8 seq 193273528361 2021-07-26T06:53:52.454 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:53:53.890 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528363 -lt 193273528361 2021-07-26T06:53:53.890 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:53.890 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-438086664199 2021-07-26T06:53:53.891 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:53.891 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:53:53.891 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-438086664199 2021-07-26T06:53:53.891 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:53.892 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=438086664199 2021-07-26T06:53:53.892 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 438086664199' 2021-07-26T06:53:53.892 INFO:tasks.workunit.client.0.smithi071.stderr:286395: waiting osd.9 seq 438086664199 2021-07-26T06:53:53.892 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:53:53.892 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 438086664200 -lt 438086664199 2021-07-26T06:53:53.893 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:53:53.893 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:53.893 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:53.893 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:53:53.894 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:53:53.894 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:53:53.894 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:53:53.894 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:53:53.894 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:53:53.895 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:53:53.895 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:53:53.895 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:53:53.895 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:53:53.896 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:53.896 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:53.896 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 5 2021-07-26T06:53:53.896 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 4 '!=' -1 2021-07-26T06:53:53.896 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:53:53.897 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=4 2021-07-26T06:53:53.897 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:53:53.897 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:53:53.897 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:53:53.898 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:53:53.898 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/ceh-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:53:53.898 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-438086664200 2021-07-26T06:53:53.898 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:53:53.898 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:53:53.899 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-438086664200 2021-07-26T06:53:53.899 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:53:53.899 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=438086664200 2021-07-26T06:53:53.899 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 438086664200' 2021-07-26T06:53:53.900 INFO:tasks.workunit.client.0.smithi071.stderr:286393: waiting osd.9 seq 438086664200 2021-07-26T06:53:53.900 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:53:53.900 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 438086664200 -lt 438086664200 2021-07-26T06:53:53.900 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:53:53.900 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:53.901 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:53.901 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:53:53.901 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:53:53.901 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:53:53.901 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:53:53.902 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:53:53.902 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:53:53.902 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:53:53.902 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:53:53.903 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:53:53.903 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:53:53.903 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:53.903 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:53.903 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 5 2021-07-26T06:53:53.904 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 4 '!=' -1 2021-07-26T06:53:53.904 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:53:53.904 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=4 2021-07-26T06:53:53.904 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:53:53.905 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:53:53.905 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:53:53.905 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:53:54.506 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:53:54.507 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:53:54.507 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:53:54.507 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:53:54.507 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:53:54.508 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:53:54.508 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:53:54.508 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:53:54.508 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:54.508 INFO:tasks.workunit.client.0.smithi071.stderr:286393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:54.509 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:53:54.509 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:53:54.509 INFO:tasks.workunit.client.0.smithi071.stderr:286393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:53:54.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:53:54.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:53:54.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 286391 2021-07-26T06:53:54.513 INFO:tasks.workunit.client.0.smithi071.stderr:phtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:53:54.513 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:53:54.514 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:53:54.514 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:53:54.514 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:53:54.514 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:53:54.515 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:53:54.515 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:53:54.515 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:53:54.515 INFO:tasks.workunit.client.0.smithi071.stderr:286395: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:53:54.516 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:53:54.516 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:53:54.516 INFO:tasks.workunit.client.0.smithi071.stderr:286395: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:53:54.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:53:54.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:53:54.517 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:53:54.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:53:54.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2021-07-26T06:53:54.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:53:54.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2021-07-26T06:53:54.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:53:54.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:53:54.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T06:53:54.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:53:54.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2021-07-26T06:53:54.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2021-07-26T06:53:54.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T06:53:54.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T06:53:54.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:53:54.864 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:53:54.864 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:53:54.864 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:53:55.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:53:22.463758+0000 2021-07-26T06:53:55.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T06:53:55.419 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.3 to repair 2021-07-26T06:53:55.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T06:53:22.463758+0000 2021-07-26T06:53:55.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T06:53:55.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:53:22.463758+0000 2021-07-26T06:53:55.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:53:55.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:53:55.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:53:55.431 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:53:55.431 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:53:55.431 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:53:55.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:53:55.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:53:55.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:53:22.463758+0000 '>' 2021-07-26T06:53:22.463758+0000 2021-07-26T06:53:55.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:53:56.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:53:56.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:53:56.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:53:56.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:53:56.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:53:56.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:53:56.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:53:56.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:53:22.463758+0000 '>' 2021-07-26T06:53:22.463758+0000 2021-07-26T06:53:56.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:53:57.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:53:57.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:53:57.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:53:57.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:53:57.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:53:57.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:53:57.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:53:58.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:53:22.463758+0000 '>' 2021-07-26T06:53:22.463758+0000 2021-07-26T06:53:58.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:53:59.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:53:59.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:53:59.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:53:59.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:53:59.249 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:53:59.249 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:53:59.249 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:53:59.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:53:22.463758+0000 '>' 2021-07-26T06:53:22.463758+0000 2021-07-26T06:53:59.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:54:00.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:54:00.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:54:00.521 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:54:00.521 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:54:00.522 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:54:00.522 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:54:00.522 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:54:00.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:53:22.463758+0000 '>' 2021-07-26T06:53:22.463758+0000 2021-07-26T06:54:00.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:54:01.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:54:01.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:54:01.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:54:01.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:54:01.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:54:01.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:54:01.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:54:02.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:53:22.463758+0000 '>' 2021-07-26T06:53:22.463758+0000 2021-07-26T06:54:02.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:54:03.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:54:03.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:54:03.078 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:54:03.078 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:54:03.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:54:03.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:54:03.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:54:03.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:53:22.463758+0000 '>' 2021-07-26T06:53:22.463758+0000 2021-07-26T06:54:03.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:54:04.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:54:04.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:54:04.354 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T06:54:04.354 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T06:54:04.355 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:54:04.355 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:54:04.355 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T06:54:04.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:54:00.581195+0000 '>' 2021-07-26T06:53:22.463758+0000 2021-07-26T06:54:04.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:54:04.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2021-07-26T06:54:04.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:54:04.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:54:04.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:54:04.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 290155"' 2021-07-26T06:54:04.630 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 290155' 2021-07-26T06:54:04.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:54:04.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T06:54:04.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T06:54:04.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 290156"' 2021-07-26T06:54:04.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 290156' 2021-07-26T06:54:04.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2021-07-26T06:54:04.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 290155 290156' 2021-07-26T06:54:04.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:54:04.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T06:54:04.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:54:04.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 290155 2021-07-26T06:54:04.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/290158: /' 2021-07-26T06:54:04.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:54:04.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/290160: /' 2021-07-26T06:54:07.322 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:54:07.323 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:54:07.323 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T06:54:07.323 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:54:07.324 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:54:07.324 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:54:07.324 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:54:07.324 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T06:54:07.325 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:54:07.325 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T06:54:07.325 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:54:07.325 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:54:07.326 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:54:07.326 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:54:07.326 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:54:07.326 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:54:07.326 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2021-07-26T06:54:07.327 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:54:07.327 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:54:07.327 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T06:54:07.327 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:54:07.328 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:54:07.328 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2021-07-26T06:54:07.328 INFO:tasks.workunit.client.0.smithi071.stderr:290158: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2021-07-26T06:54:07.328 INFO:tasks.workunit.client.0.smithi071.stderr:290158: Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T06:54:07.329 INFO:tasks.workunit.client.0.smithi071.stderr:290158: _ 2021-07-26T06:54:07.329 INFO:tasks.workunit.client.0.smithi071.stderr:290158: hinfo_key 2021-07-26T06:54:07.329 INFO:tasks.workunit.client.0.smithi071.stderr:290158: snapset 2021-07-26T06:54:07.330 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T06:54:07.330 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:54:07.331 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:54:07.331 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:54:07.331 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:54:07.331 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T06:54:07.332 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:54:07.332 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:54:07.333 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:54:07.333 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:54:07.333 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T06:54:07.333 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T06:54:07.333 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:54:07.334 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:54:07.334 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:54:07.334 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:54:07.334 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:54:07.335 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:54:07.335 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:54:07.335 INFO:tasks.workunit.client.0.smithi071.stderr:290158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:54:07.335 INFO:tasks.workunit.client.0.smithi071.stderr:290158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:54:07.335 INFO:tasks.workunit.client.0.smithi071.stderr:290158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:54:07.336 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:54:07.336 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:54:07.336 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:54:07.336 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:54:07.337 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:54:07.337 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:54:07.337 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:54:07.337 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:54:07.337 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:54:07.338 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:54:07.338 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T06:54:07.338 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:54:07.338 INFO:tasks.workunit.client.0.smithi071.stderr:290158: start osd.3 2021-07-26T06:54:07.584 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-f290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:54:07.584 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:54:07.585 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=5 2021-07-26T06:54:07.585 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:54:07.585 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:54:07.585 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:54:07.586 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:54:07.586 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=5 2021-07-26T06:54:07.586 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:54:07.586 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2021-07-26T06:54:07.586 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:54:07.587 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:54:07.587 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:54:07.587 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:54:07.588 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:54:07.588 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:54:07.588 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2021-07-26T06:54:07.588 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:54:07.589 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:54:07.589 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=5 2021-07-26T06:54:07.590 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:54:07.590 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:54:07.590 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2021-07-26T06:54:07.591 INFO:tasks.workunit.client.0.smithi071.stderr:290160: Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:54:07.591 INFO:tasks.workunit.client.0.smithi071.stderr:290160: Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T06:54:07.591 INFO:tasks.workunit.client.0.smithi071.stderr:290160: _ 2021-07-26T06:54:07.591 INFO:tasks.workunit.client.0.smithi071.stderr:290160: hinfo_key 2021-07-26T06:54:07.592 INFO:tasks.workunit.client.0.smithi071.stderr:290160: snapset 2021-07-26T06:54:07.592 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2021-07-26T06:54:07.592 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:54:07.592 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:54:07.592 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=5 2021-07-26T06:54:07.593 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:54:07.593 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/5 2021-07-26T06:54:07.593 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:54:07.594 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:54:07.594 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:54:07.594 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:54:07.595 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2021-07-26T06:54:07.595 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2021-07-26T06:54:07.595 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:54:07.596 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:54:07.596 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:54:07.596 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:54:07.596 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:54:07.597 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:54:07.597 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:54:07.597 INFO:tasks.workunit.client.0.smithi071.stderr:290160: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:54:07.597 INFO:tasks.workunit.client.0.smithi071.stderr:290160: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:54:07.597 INFO:tasks.workunit.client.0.smithi071.stderr:290160: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:54:07.598 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:54:07.598 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:54:07.598 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:54:07.598 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:54:07.599 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:54:07.599 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:54:07.599 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:54:07.599 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:54:07.600 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:54:07.600 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:54:07.600 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/5 2021-07-26T06:54:07.600 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.5 2021-07-26T06:54:07.600 INFO:tasks.workunit.client.0.smithi071.stderr:290160: start osd.5 2021-07-26T06:54:11.521 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 5 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scruile=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:54:11.522 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T06:54:11.522 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:54:11.522 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:54:11.522 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:54:11.523 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:54:11.523 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 2021-07-26T06:54:07.349+0000 7f3f20b2d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:11.523 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 2021-07-26T06:54:07.351+0000 7f3f20b2d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:11.523 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 2021-07-26T06:54:07.352+0000 7f3f20b2d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:11.524 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:54:11.524 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:54:11.524 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:54:11.524 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:54:11.525 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:54:11.525 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:11.525 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:54:11.525 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 0 2021-07-26T06:54:11.525 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:11.526 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:54:11.526 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 2021-07-26T06:54:07.870+0000 7f3f20b2d0c0 -1 Falling back to public interface 2021-07-26T06:54:11.526 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:11.526 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:11.527 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:11.527 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:54:11.527 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 1 2021-07-26T06:54:11.527 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:11.528 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:54:11.528 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:11.528 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:11.528 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:11.528 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:54:11.529 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 2 2021-07-26T06:54:11.529 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:11.529 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:54:11.529 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:11.530 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 2021-07-26T06:54:11.078+0000 7f3f20b2d0c0 -1 osd.3 110 log_to_monitors {default=true} 2021-07-26T06:54:11.530 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:11.530 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:11.530 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:54:11.531 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 3 2021-07-26T06:54:11.531 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:11.799 INFO:tasks.workunit.client.0.smithi071.stderr:290158b-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:54:11.799 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/5/whoami 2021-07-26T06:54:11.799 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 5 = 5 ']' 2021-07-26T06:54:11.799 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:54:11.800 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:54:11.800 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:54:11.800 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 2021-07-26T06:54:07.608+0000 7f268867b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:11.800 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 2021-07-26T06:54:07.610+0000 7f268867b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:11.801 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 2021-07-26T06:54:07.611+0000 7f268867b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:11.801 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 5 2021-07-26T06:54:11.801 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:54:11.801 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T06:54:11.801 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:54:11.802 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:54:11.802 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:11.802 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:54:11.802 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 0 2021-07-26T06:54:11.802 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:11.803 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:54:11.803 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 2021-07-26T06:54:08.129+0000 7f268867b0c0 -1 Falling back to public interface 2021-07-26T06:54:11.803 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:11.803 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:11.804 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:11.804 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:54:11.804 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 1 2021-07-26T06:54:11.804 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:11.804 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:54:11.805 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:11.805 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:11.805 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:11.805 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:54:11.805 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 2 2021-07-26T06:54:11.806 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:11.806 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:54:11.806 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:11.806 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 2021-07-26T06:54:11.436+0000 7f268867b0c0 -1 osd.5 110 log_to_monitors {default=true} 2021-07-26T06:54:11.807 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:11.807 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:11.807 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:54:11.807 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 3 2021-07-26T06:54:11.807 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:12.927 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:54:12.927 INFO:tasks.workunit.client.0.smithi071.stderr:290158: osd.3 up in weight 1 up_from 114 up_thru 109 down_at 111 last_clean_interval [109,110) [v2:127.0.0.1:6826/290963,v1:127.0.0.1:6827/290963] [v2:127.0.0.1:6828/290963,v1:127.0.0.1:6829/290963] exists,up 9c55da62-886d-4a69-a717-2a90c8e238a9 2021-07-26T06:54:12.928 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:54:12.928 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:54:12.928 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:54:12.928 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:54:12.929 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:54:12.929 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:54:12.929 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:54:12.929 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:54:12.929 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:54:12.930 INFO:tasks.workunit.client.0.smithi071.stderr:290158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:54:12.930 INFO:tasks.workunit.client.0.smithi071.stderr:290158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:54:12.930 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:54:12.930 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:54:12.931 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:54:12.931 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:54:12.931 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:54:12.931 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:54:12.932 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:54:12.932 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:54:12.932 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:54:12.932 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 1 2021-07-26T06:54:12.932 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 2 2021-07-26T06:54:12.933 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 3 2021-07-26T06:54:12.933 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 4 2021-07-26T06:54:12.933 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 5 2021-07-26T06:54:12.933 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 6 2021-07-26T06:54:12.934 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 7 2021-07-26T06:54:12.934 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 8 2021-07-26T06:54:12.934 INFO:tasks.workunit.client.0.smithi071.stderr:290158: 9' 2021-07-26T06:54:12.934 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:54:12.934 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:12.935 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:54:12.935 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836546 2021-07-26T06:54:12.935 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836546 2021-07-26T06:54:12.935 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836546' 2021-07-26T06:54:12.935 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:12.936 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:54:12.936 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673024 2021-07-26T06:54:12.936 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673024 2021-07-26T06:54:12.936 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836546 1-42949673024' 2021-07-26T06:54:13.312 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cecephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T06:54:13.313 INFO:tasks.workunit.client.0.smithi071.stderr:290160: osd.5 up in weight 1 up_from 115 up_thru 109 down_at 111 last_clean_interval [109,110) [v2:127.0.0.1:6842/291012,v1:127.0.0.1:6843/291012] [v2:127.0.0.1:6844/291012,v1:127.0.0.1:6845/291012] exists,up bbee9dc4-1bba-496c-96c5-7f58e2aee6bc 2021-07-26T06:54:13.313 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:54:13.313 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:54:13.313 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:54:13.314 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:54:13.314 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:54:13.314 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:54:13.314 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:54:13.315 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:54:13.315 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:54:13.315 INFO:tasks.workunit.client.0.smithi071.stderr:290160: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:54:13.315 INFO:tasks.workunit.client.0.smithi071.stderr:290160: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:54:13.315 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:54:13.316 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:54:13.316 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:54:13.316 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:54:13.316 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:54:13.317 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:54:13.317 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:54:13.317 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:54:13.317 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:54:13.317 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 1 2021-07-26T06:54:13.318 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 2 2021-07-26T06:54:13.318 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 3 2021-07-26T06:54:13.318 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 4 2021-07-26T06:54:13.319 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 5 2021-07-26T06:54:13.319 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 6 2021-07-26T06:54:13.319 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 7 2021-07-26T06:54:13.319 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 8 2021-07-26T06:54:13.320 INFO:tasks.workunit.client.0.smithi071.stderr:290160: 9' 2021-07-26T06:54:13.320 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:54:13.320 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:13.320 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:54:13.320 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836547 2021-07-26T06:54:13.321 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836547 2021-07-26T06:54:13.321 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836547' 2021-07-26T06:54:13.321 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:13.322 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:54:13.322 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673025 2021-07-26T06:54:13.322 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673025 2021-07-26T06:54:13.322 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836547 1-42949673025' 2021-07-26T06:54:14.162 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.clphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:14.163 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:54:14.163 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509502 2021-07-26T06:54:14.163 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509502 2021-07-26T06:54:14.163 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836546 1-42949673024 2-64424509502' 2021-07-26T06:54:14.164 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:14.164 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:54:14.164 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=489626271746 2021-07-26T06:54:14.164 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 489626271746 2021-07-26T06:54:14.165 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836546 1-42949673024 2-64424509502 3-489626271746' 2021-07-26T06:54:14.165 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:14.165 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:54:14.165 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182457 2021-07-26T06:54:14.165 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182457 2021-07-26T06:54:14.166 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836546 1-42949673024 2-64424509502 3-489626271746 4-107374182457' 2021-07-26T06:54:14.166 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:14.166 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:54:14.166 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=493921239042 2021-07-26T06:54:14.166 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 493921239042 2021-07-26T06:54:14.167 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836546 1-42949673024 2-64424509502 3-489626271746 4-107374182457 5-493921239042' 2021-07-26T06:54:14.167 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:14.167 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:54:14.167 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855413 2021-07-26T06:54:14.168 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855413 2021-07-26T06:54:14.168 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836546 1-42949673024 2-64424509502 3-489626271746 4-107374182457 5-493921239042 6-150323855413' 2021-07-26T06:54:14.168 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:14.168 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:54:14.168 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691891 2021-07-26T06:54:14.169 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691891 2021-07-26T06:54:14.169 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836546 1-42949673024 2-64424509502 3-489626271746 4-107374182457 5-493921239042 6-150323855413 7-171798691891' 2021-07-26T06:54:14.169 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:14.441 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephient.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:14.441 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:54:14.441 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509503 2021-07-26T06:54:14.442 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509503 2021-07-26T06:54:14.442 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836547 1-42949673025 2-64424509503' 2021-07-26T06:54:14.442 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:14.442 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:54:14.443 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=489626271747 2021-07-26T06:54:14.443 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 489626271747 2021-07-26T06:54:14.443 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836547 1-42949673025 2-64424509503 3-489626271747' 2021-07-26T06:54:14.444 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:14.444 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:54:14.444 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182458 2021-07-26T06:54:14.444 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182458 2021-07-26T06:54:14.444 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836547 1-42949673025 2-64424509503 3-489626271747 4-107374182458' 2021-07-26T06:54:14.445 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:14.445 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T06:54:14.445 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=493921239043 2021-07-26T06:54:14.445 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 493921239043 2021-07-26T06:54:14.446 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836547 1-42949673025 2-64424509503 3-489626271747 4-107374182458 5-493921239043' 2021-07-26T06:54:14.446 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:14.446 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-07-26T06:54:14.446 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855414 2021-07-26T06:54:14.446 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855414 2021-07-26T06:54:14.447 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836547 1-42949673025 2-64424509503 3-489626271747 4-107374182458 5-493921239043 6-150323855414' 2021-07-26T06:54:14.447 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:14.447 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-07-26T06:54:14.447 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691892 2021-07-26T06:54:14.448 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691892 2021-07-26T06:54:14.448 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836547 1-42949673025 2-64424509503 3-489626271747 4-107374182458 5-493921239043 6-150323855414 7-171798691892' 2021-07-26T06:54:14.448 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:15.065 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.clietest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:54:15.065 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528368 2021-07-26T06:54:15.065 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528368 2021-07-26T06:54:15.065 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836546 1-42949673024 2-64424509502 3-489626271746 4-107374182457 5-493921239042 6-150323855413 7-171798691891 8-193273528368' 2021-07-26T06:54:15.066 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:15.066 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:54:15.066 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=438086664206 2021-07-26T06:54:15.066 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 438086664206 2021-07-26T06:54:15.067 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836546 1-42949673024 2-64424509502 3-489626271746 4-107374182457 5-493921239042 6-150323855413 7-171798691891 8-193273528368 9-438086664206' 2021-07-26T06:54:15.067 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:15.067 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836546 2021-07-26T06:54:15.067 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:15.068 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:54:15.068 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836546 2021-07-26T06:54:15.068 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:15.068 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836546 2021-07-26T06:54:15.069 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2021-07-26T06:54:15.069 INFO:tasks.workunit.client.0.smithi071.stderr:290158: waiting osd.0 seq 21474836546 2021-07-26T06:54:15.069 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:54:15.069 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836547 -lt 21474836546 2021-07-26T06:54:15.069 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:15.070 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673024 2021-07-26T06:54:15.070 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:15.070 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:54:15.070 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673024 2021-07-26T06:54:15.071 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:15.071 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673024 2021-07-26T06:54:15.071 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673024' 2021-07-26T06:54:15.071 INFO:tasks.workunit.client.0.smithi071.stderr:290158: waiting osd.1 seq 42949673024 2021-07-26T06:54:15.072 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:54:15.072 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673025 -lt 42949673024 2021-07-26T06:54:15.072 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:15.072 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509502 2021-07-26T06:54:15.072 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:15.316 INFO:tasks.workunit.client.0.smithi071.stderr:290nt.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-07-26T06:54:15.316 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528369 2021-07-26T06:54:15.316 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528369 2021-07-26T06:54:15.316 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836547 1-42949673025 2-64424509503 3-489626271747 4-107374182458 5-493921239043 6-150323855414 7-171798691892 8-193273528369' 2021-07-26T06:54:15.317 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:15.317 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-07-26T06:54:15.317 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=438086664207 2021-07-26T06:54:15.317 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 438086664207 2021-07-26T06:54:15.318 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836547 1-42949673025 2-64424509503 3-489626271747 4-107374182458 5-493921239043 6-150323855414 7-171798691892 8-193273528369 9-438086664207' 2021-07-26T06:54:15.318 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:15.318 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836547 2021-07-26T06:54:15.319 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:15.319 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:54:15.319 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836547 2021-07-26T06:54:15.319 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836547 2021-07-26T06:54:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2021-07-26T06:54:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:290160: waiting osd.0 seq 21474836547 2021-07-26T06:54:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:54:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836547 -lt 21474836547 2021-07-26T06:54:15.321 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:15.321 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673025 2021-07-26T06:54:15.321 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:15.321 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:54:15.322 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673025 2021-07-26T06:54:15.322 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:15.322 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673025 2021-07-26T06:54:15.322 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673025' 2021-07-26T06:54:15.323 INFO:tasks.workunit.client.0.smithi071.stderr:290160: waiting osd.1 seq 42949673025 2021-07-26T06:54:15.323 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:54:15.323 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673025 -lt 42949673025 2021-07-26T06:54:15.323 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:15.323 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509503 2021-07-26T06:54:15.324 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:15.980 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubun158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:54:15.981 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509502 2021-07-26T06:54:15.981 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:15.981 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509502 2021-07-26T06:54:15.981 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509502' 2021-07-26T06:54:15.981 INFO:tasks.workunit.client.0.smithi071.stderr:290158: waiting osd.2 seq 64424509502 2021-07-26T06:54:15.982 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:54:15.982 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509504 -lt 64424509502 2021-07-26T06:54:15.982 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:15.982 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-489626271746 2021-07-26T06:54:15.983 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:15.983 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:54:15.983 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-489626271746 2021-07-26T06:54:15.983 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:15.983 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=489626271746 2021-07-26T06:54:15.984 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 489626271746' 2021-07-26T06:54:15.984 INFO:tasks.workunit.client.0.smithi071.stderr:290158: waiting osd.3 seq 489626271746 2021-07-26T06:54:15.984 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:54:15.984 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 489626271747 -lt 489626271746 2021-07-26T06:54:15.985 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:15.985 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182457 2021-07-26T06:54:15.985 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:15.985 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:54:15.985 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182457 2021-07-26T06:54:15.986 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:15.986 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182457 2021-07-26T06:54:15.986 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182457' 2021-07-26T06:54:15.986 INFO:tasks.workunit.client.0.smithi071.stderr:290158: waiting osd.4 seq 107374182457 2021-07-26T06:54:15.986 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:54:15.987 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182458 -lt 107374182457 2021-07-26T06:54:15.987 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:15.987 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-493921239042 2021-07-26T06:54:15.987 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:15.988 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:54:15.988 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-493921239042 2021-07-26T06:54:15.988 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:16.222 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flustu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:54:16.222 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509503 2021-07-26T06:54:16.223 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:16.223 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509503 2021-07-26T06:54:16.223 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2021-07-26T06:54:16.223 INFO:tasks.workunit.client.0.smithi071.stderr:290160: waiting osd.2 seq 64424509503 2021-07-26T06:54:16.224 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:54:16.224 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509504 -lt 64424509503 2021-07-26T06:54:16.224 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:16.224 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-489626271747 2021-07-26T06:54:16.225 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:16.225 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:54:16.225 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-489626271747 2021-07-26T06:54:16.225 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:16.225 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=489626271747 2021-07-26T06:54:16.226 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 489626271747' 2021-07-26T06:54:16.226 INFO:tasks.workunit.client.0.smithi071.stderr:290160: waiting osd.3 seq 489626271747 2021-07-26T06:54:16.226 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:54:16.226 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 489626271747 -lt 489626271747 2021-07-26T06:54:16.227 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:16.227 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182458 2021-07-26T06:54:16.227 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:16.227 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:54:16.228 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182458 2021-07-26T06:54:16.228 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:16.228 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182458 2021-07-26T06:54:16.228 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182458' 2021-07-26T06:54:16.228 INFO:tasks.workunit.client.0.smithi071.stderr:290160: waiting osd.4 seq 107374182458 2021-07-26T06:54:16.229 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:54:16.229 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182458 -lt 107374182458 2021-07-26T06:54:16.229 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:16.229 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-493921239043 2021-07-26T06:54:16.229 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:16.230 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T06:54:16.230 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-493921239043 2021-07-26T06:54:16.230 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:17.181 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seh_pg_stats: seq=493921239042 2021-07-26T06:54:17.181 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 493921239042' 2021-07-26T06:54:17.181 INFO:tasks.workunit.client.0.smithi071.stderr:290158: waiting osd.5 seq 493921239042 2021-07-26T06:54:17.182 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:54:17.182 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 493921239043 -lt 493921239042 2021-07-26T06:54:17.182 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:17.182 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855413 2021-07-26T06:54:17.182 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:17.183 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:54:17.183 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855413 2021-07-26T06:54:17.183 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:17.183 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855413 2021-07-26T06:54:17.184 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855413' 2021-07-26T06:54:17.184 INFO:tasks.workunit.client.0.smithi071.stderr:290158: waiting osd.6 seq 150323855413 2021-07-26T06:54:17.184 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:54:17.184 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855414 -lt 150323855413 2021-07-26T06:54:17.185 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:17.185 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691891 2021-07-26T06:54:17.185 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:17.185 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:54:17.185 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691891 2021-07-26T06:54:17.186 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:17.186 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691891 2021-07-26T06:54:17.186 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691891' 2021-07-26T06:54:17.186 INFO:tasks.workunit.client.0.smithi071.stderr:290158: waiting osd.7 seq 171798691891 2021-07-26T06:54:17.187 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:54:17.187 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691893 -lt 171798691891 2021-07-26T06:54:17.187 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:17.187 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528368 2021-07-26T06:54:17.187 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:17.188 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:54:17.188 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528368 2021-07-26T06:54:17.188 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:17.188 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528368 2021-07-26T06:54:17.189 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528368' 2021-07-26T06:54:17.189 INFO:tasks.workunit.client.0.smithi071.stderr:290158: waiting osd.8 seq 193273528368 2021-07-26T06:54:17.189 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:54:17.429 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flq=493921239043 2021-07-26T06:54:17.429 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 493921239043' 2021-07-26T06:54:17.429 INFO:tasks.workunit.client.0.smithi071.stderr:290160: waiting osd.5 seq 493921239043 2021-07-26T06:54:17.429 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T06:54:17.430 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 493921239043 -lt 493921239043 2021-07-26T06:54:17.430 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:17.430 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 6-150323855414 2021-07-26T06:54:17.430 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:17.431 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=6 2021-07-26T06:54:17.431 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 6-150323855414 2021-07-26T06:54:17.431 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:17.431 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855414 2021-07-26T06:54:17.432 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.6 seq 150323855414' 2021-07-26T06:54:17.432 INFO:tasks.workunit.client.0.smithi071.stderr:290160: waiting osd.6 seq 150323855414 2021-07-26T06:54:17.432 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 6 2021-07-26T06:54:17.432 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855414 -lt 150323855414 2021-07-26T06:54:17.432 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:17.433 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 7-171798691892 2021-07-26T06:54:17.433 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:17.433 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=7 2021-07-26T06:54:17.433 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 7-171798691892 2021-07-26T06:54:17.434 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:17.434 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691892 2021-07-26T06:54:17.434 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2021-07-26T06:54:17.434 INFO:tasks.workunit.client.0.smithi071.stderr:290160: waiting osd.7 seq 171798691892 2021-07-26T06:54:17.434 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 7 2021-07-26T06:54:17.435 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691893 -lt 171798691892 2021-07-26T06:54:17.435 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:17.435 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 8-193273528369 2021-07-26T06:54:17.435 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:17.435 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=8 2021-07-26T06:54:17.436 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 8-193273528369 2021-07-26T06:54:17.436 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:17.436 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528369 2021-07-26T06:54:17.436 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2021-07-26T06:54:17.437 INFO:tasks.workunit.client.0.smithi071.stderr:290160: waiting osd.8 seq 193273528369 2021-07-26T06:54:17.437 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 8 2021-07-26T06:54:18.451 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ush_pg_stats: test 193273528370 -lt 193273528368 2021-07-26T06:54:18.451 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:18.451 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-438086664206 2021-07-26T06:54:18.452 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:18.452 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:54:18.452 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-438086664206 2021-07-26T06:54:18.452 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:18.453 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=438086664206 2021-07-26T06:54:18.453 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 438086664206' 2021-07-26T06:54:18.453 INFO:tasks.workunit.client.0.smithi071.stderr:290158: waiting osd.9 seq 438086664206 2021-07-26T06:54:18.453 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:54:18.454 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 438086664207 -lt 438086664206 2021-07-26T06:54:18.454 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:54:18.454 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:54:18.454 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:54:18.454 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:54:18.455 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:54:18.455 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:54:18.455 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:54:18.455 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:54:18.456 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:54:18.456 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:54:18.456 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:54:18.456 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:54:18.456 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:54:18.457 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:54:18.457 INFO:tasks.workunit.client.0.smithi071.stderr:290158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:54:18.457 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:54:18.457 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:54:18.458 INFO:tasks.workunit.client.0.smithi071.stderr:290158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:54:18.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:54:18.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T06:54:18.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 290156 2021-07-26T06:54:18.703 INFO:tasks.workunit.client.0.smithi071.stderr:test 193273528370 -lt 193273528369 2021-07-26T06:54:18.703 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:18.703 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 9-438086664207 2021-07-26T06:54:18.704 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:18.704 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=9 2021-07-26T06:54:18.704 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 9-438086664207 2021-07-26T06:54:18.704 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:18.705 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=438086664207 2021-07-26T06:54:18.705 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.9 seq 438086664207' 2021-07-26T06:54:18.705 INFO:tasks.workunit.client.0.smithi071.stderr:290160: waiting osd.9 seq 438086664207 2021-07-26T06:54:18.705 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 9 2021-07-26T06:54:18.705 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 438086664207 -lt 438086664207 2021-07-26T06:54:18.706 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:54:18.706 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:54:18.706 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:54:18.706 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T06:54:18.706 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:54:18.707 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:54:18.707 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:54:18.707 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:54:18.707 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:54:18.708 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:54:18.708 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:54:18.708 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T06:54:18.708 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:54:18.708 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:54:18.709 INFO:tasks.workunit.client.0.smithi071.stderr:290160: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:54:18.709 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T06:54:18.709 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:54:18.709 INFO:tasks.workunit.client.0.smithi071.stderr:290160: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:54:18.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T06:54:18.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T06:54:18.710 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T06:54:18.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T06:54:18.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2021-07-26T06:54:18.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2021-07-26T06:54:18.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:54:18.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:54:18.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:54:18.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:54:18.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:54:18.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:54:18.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:54:18.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:54:18.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:54:18.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:54:18.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:54:18.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:54:18.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:54:18.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:54:18.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:54:18.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:54:18.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:54:18.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:54:18.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:54:18.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:54:18.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:54:18.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:54:18.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:54:18.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:54:18.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:54:18.892 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:54:18.892 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:54:18.892 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:54:18.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:54:18.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:54:18.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:54:18.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:54:18.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:54:18.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:54:18.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:54:18.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:54:18.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:54:18.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:54:18.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:54:18.897 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:54:18.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:54:18.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:54:18.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:54:18.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:54:18.899 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:54:18.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:54:18.899 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:54:18.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:54:18.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:54:18.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:54:18.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:54:18.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:54:18.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:54:18.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:54:18.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:54:18.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:54:18.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:54:18.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:54:18.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:54:18.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:54:18.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:54:18.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:54:18.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:54:18.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:54:18.906 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:54:18.906 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:54:18.906 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:54:18.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:54:18.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:54:18.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:54:18.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:54:18.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:54:18.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2021-07-26T06:54:18.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:93: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2021-07-26T06:54:18.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local poolname=rbd 2021-07-26T06:54:18.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:96: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2021-07-26T06:54:18.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:54:18.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:54:18.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:54:18.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:54:18.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:54:18.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2021-07-26T06:54:19.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:54:19.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:54:19.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:54:19.266 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:54:19.266 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:54:19.267 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:54:19.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:54:19.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2021-07-26T06:54:19.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:54:19.526 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:54:19.526 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:54:19.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:54:19.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:54:19.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:54:19.527 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:54:19.527 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:54:19.528 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:54:19.528 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:54:19.528 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:54:19.528 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:54:19.529 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:54:19.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:54:19.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:54:19.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:54:19.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:54:19.623 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:54:19.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:54:19.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:54:19.624 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:54:19.624 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:54:19.625 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:54:19.625 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:54:19.625 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:54:19.625 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:54:19.626 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:54:19.626 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:54:19.626 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:54:19.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2021-07-26T06:54:19.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:54:19.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:54:19.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:54:19.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:54:19.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:54:19.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:54:20.553 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:54:20.553 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:54:20.553 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:54:20.553 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:54:20.554 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:54:20.554 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:54:20.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:54:20.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:54:20.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:54:20.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2021-07-26T06:54:20.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:54:20.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:54:20.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:54:20.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:54:20.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:54:20.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:54:20.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:54:20.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:54:20.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:54:20.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:54:20.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:54:20.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:54:20.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:54:20.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:54:20.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:54:20.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:54:20.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:54:20.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:54:20.585 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:54:20.585 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:54:20.586 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:54:20.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:54:20.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:54:20.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:54:20.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:54:20.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:54:20.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:54:20.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:54:20.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:54:20.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:54:20.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:54:20.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:54:20.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:54:20.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:54:20.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:54:20.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=496c3c79-e98d-4917-9c9c-bb9178d538c5 2021-07-26T06:54:20.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 496c3c79-e98d-4917-9c9c-bb9178d538c5' 2021-07-26T06:54:20.590 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 496c3c79-e98d-4917-9c9c-bb9178d538c5 2021-07-26T06:54:20.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:54:20.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAcXP5gGHxyIxAAcT3D23cgimA5pwuWHxmOQA== 2021-07-26T06:54:20.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAcXP5gGHxyIxAAcT3D23cgimA5pwuWHxmOQA=="}' 2021-07-26T06:54:20.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 496c3c79-e98d-4917-9c9c-bb9178d538c5 -i td/osd-scrub-repair/0/new.json 2021-07-26T06:54:20.850 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:54:20.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:54:20.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAcXP5gGHxyIxAAcT3D23cgimA5pwuWHxmOQA== --osd-uuid 496c3c79-e98d-4917-9c9c-bb9178d538c5 2021-07-26T06:54:20.886 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:20.886+0000 7f132880d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:20.887 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:20.887+0000 7f132880d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:20.888 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:20.888+0000 7f132880d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:20.889 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:20.889+0000 7f132880d0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:54:20.889 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:20.889+0000 7f132880d0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:54:20.889 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:20.889+0000 7f132880d0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:54:20.942 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:20.941+0000 7f132880d0c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T06:54:26.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:54:26.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:54:26.579 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:54:26.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:54:26.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:54:26.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:54:26.911 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:54:26.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:54:26.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:54:26.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:54:26.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:54:26.933 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:26.932+0000 7f002948d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:26.934 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:26.934+0000 7f002948d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:26.935 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:26.935+0000 7f002948d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:27.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:54:27.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:54:27.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:54:27.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:54:27.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:54:27.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:27.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:54:27.216 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:54:27.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:27.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:54:27.451 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:27.451+0000 7f002948d0c0 -1 Falling back to public interface 2021-07-26T06:54:27.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:28.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:28.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:28.526 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:54:28.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:54:28.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:28.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:54:28.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:29.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:29.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:29.801 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:54:29.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:54:29.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:29.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:54:29.807 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:29.807+0000 7f002948d0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:54:30.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:31.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:31.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:31.083 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:54:31.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:54:31.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:31.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:54:31.362 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/294747,v1:127.0.0.1:6803/294747] [v2:127.0.0.1:6804/294747,v1:127.0.0.1:6805/294747] exists,up 496c3c79-e98d-4917-9c9c-bb9178d538c5 2021-07-26T06:54:31.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:54:31.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:54:31.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:54:31.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2021-07-26T06:54:31.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:54:31.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:54:31.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:54:31.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:54:31.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:54:31.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:54:31.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:54:31.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:54:31.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:54:31.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:54:31.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:54:31.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:54:31.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:54:31.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:54:31.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:54:31.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:54:31.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:54:31.373 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:54:31.373 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:54:31.373 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:54:31.373 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:54:31.374 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:54:31.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:54:31.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:54:31.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:54:31.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:54:31.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:54:31.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:54:31.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:54:31.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:54:31.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:54:31.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:54:31.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:54:31.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:54:31.378 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:54:31.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d81b7e7c-7669-4e02-a5b5-3a8031948699 2021-07-26T06:54:31.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d81b7e7c-7669-4e02-a5b5-3a8031948699' 2021-07-26T06:54:31.379 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 d81b7e7c-7669-4e02-a5b5-3a8031948699 2021-07-26T06:54:31.379 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:54:31.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAnXP5gRlmjFhAAPEX6Cu1bAReLg8U5jF71uw== 2021-07-26T06:54:31.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAnXP5gRlmjFhAAPEX6Cu1bAReLg8U5jF71uw=="}' 2021-07-26T06:54:31.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d81b7e7c-7669-4e02-a5b5-3a8031948699 -i td/osd-scrub-repair/1/new.json 2021-07-26T06:54:31.809 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:54:31.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:54:31.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAnXP5gRlmjFhAAPEX6Cu1bAReLg8U5jF71uw== --osd-uuid d81b7e7c-7669-4e02-a5b5-3a8031948699 2021-07-26T06:54:31.846 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:31.845+0000 7fb56a1af0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:31.848 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:31.847+0000 7fb56a1af0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:31.848 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:31.848+0000 7fb56a1af0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:31.849 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:31.849+0000 7fb56a1af0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:54:31.849 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:31.849+0000 7fb56a1af0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:54:31.850 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:31.849+0000 7fb56a1af0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:54:31.926 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:31.926+0000 7fb56a1af0c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T06:54:37.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:54:37.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:54:37.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:54:37.345 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:54:37.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:54:37.682 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:54:37.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:54:37.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:54:37.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:54:37.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:54:37.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:54:37.704 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:37.704+0000 7f145e7ef0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:37.724 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:37.724+0000 7f145e7ef0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:37.725 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:37.725+0000 7f145e7ef0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:37.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:54:37.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:54:37.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:54:37.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:54:37.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:54:37.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:37.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:54:37.957 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:54:37.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:37.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:54:38.241 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:38.241+0000 7f145e7ef0c0 -1 Falling back to public interface 2021-07-26T06:54:38.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:39.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:39.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:39.260 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:54:39.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:54:39.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:39.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:54:39.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:40.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:40.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:40.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:54:40.538 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:54:40.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:40.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:54:40.578 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:40.577+0000 7f145e7ef0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:54:40.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:41.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:41.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:41.910 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:54:41.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:54:41.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:41.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:54:42.195 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/296146,v1:127.0.0.1:6811/296146] [v2:127.0.0.1:6812/296146,v1:127.0.0.1:6813/296146] exists,up d81b7e7c-7669-4e02-a5b5-3a8031948699 2021-07-26T06:54:42.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:54:42.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:54:42.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:54:42.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: create_rbd_pool 2021-07-26T06:54:42.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:54:42.455 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T06:54:42.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:54:42.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:54:42.890 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T06:54:42.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:54:43.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:54:45.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: wait_for_clean 2021-07-26T06:54:45.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:54:45.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:54:45.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:54:45.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:54:45.405 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:54:45.406 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:54:45.407 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:54:45.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:54:45.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:54:45.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:54:45.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:54:45.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:54:45.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:54:45.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:54:45.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:54:45.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:54:45.780 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T06:54:45.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:54:45.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:45.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:54:45.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836484 2021-07-26T06:54:45.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836484 2021-07-26T06:54:45.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484' 2021-07-26T06:54:45.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:45.933 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:54:46.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672962 2021-07-26T06:54:46.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672962 2021-07-26T06:54:46.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484 1-42949672962' 2021-07-26T06:54:46.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:46.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836484 2021-07-26T06:54:46.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:46.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:54:46.052 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836484 2021-07-26T06:54:46.052 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:46.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836484 2021-07-26T06:54:46.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2021-07-26T06:54:46.053 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836484 2021-07-26T06:54:46.053 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:54:46.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836483 -lt 21474836484 2021-07-26T06:54:46.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:54:47.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:54:47.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:54:47.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836485 -lt 21474836484 2021-07-26T06:54:47.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:47.595 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672962 2021-07-26T06:54:47.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:47.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:54:47.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672962 2021-07-26T06:54:47.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:47.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672962 2021-07-26T06:54:47.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2021-07-26T06:54:47.598 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672962 2021-07-26T06:54:47.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:54:47.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672963 -lt 42949672962 2021-07-26T06:54:47.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:54:47.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:54:47.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:54:48.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:54:48.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:54:48.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:54:48.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:54:48.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:54:48.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:54:48.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:54:48.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:54:48.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:54:48.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:54:48.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:54:48.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:54:48.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:54:48.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:54:48.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:54:48.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:103: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2021-07-26T06:54:48.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:54:48.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=rbd 2021-07-26T06:54:48.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2021-07-26T06:54:48.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:54:48.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:54:48.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:54:49.292 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:54:49.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:54:49.710 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:54:49.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:54:49.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:54:49.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T06:54:49.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2021-07-26T06:54:49.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=rbd 2021-07-26T06:54:49.963 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T06:54:49.963 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary rbd SOMETHING 2021-07-26T06:54:49.963 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=rbd 2021-07-26T06:54:49.964 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:54:49.964 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map rbd SOMETHING 2021-07-26T06:54:49.964 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:54:50.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:54:50.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map rbd SOMETHING 2021-07-26T06:54:50.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:54:50.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2021-07-26T06:54:50.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2021-07-26T06:54:50.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=rbd 2021-07-26T06:54:50.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=0 2021-07-26T06:54:50.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:54:50.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:54:50.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:54:50.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:54:50.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:54:50.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:54:50.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:54:50.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:54:50.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:54:50.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:54:50.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:54:50.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:54:50.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:54:50.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:54:50.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:54:50.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:54:50.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:54:50.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T06:54:50.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:54:50.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:54:50.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:54:50.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:54:50.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:54:50.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2021-07-26T06:54:52.387 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:54:52.388 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:54:52.388 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:54:52.388 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:54:52.389 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:eb822e21:::SOMETHING:head# 2021-07-26T06:54:52.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T06:54:52.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:54:52.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:54:52.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:54:52.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:54:52.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:54:52.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:54:52.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:54:52.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:54:52.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:54:52.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:54:52.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:54:52.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:54:52.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:54:52.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:54:52.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:54:52.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:54:52.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:54:52.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:54:52.913 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:54:52.914 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:54:52.914 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:54:52.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:54:52.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:54:52.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:54:52.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:54:52.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:54:52.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:54:52.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:54:52.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:54:52.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:54:52.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:54:52.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:54:52.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:54:52.917 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:54:52.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:54:52.917 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:54:52.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:54:52.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:54:52.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:54:52.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:54:52.932 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:52.931+0000 7f307ec160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:52.934 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:52.933+0000 7f307ec160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:52.935 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:52.934+0000 7f307ec160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:54:53.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:54:53.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:54:53.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:54:53.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:54:53.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:54:53.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:53.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:54:53.189 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:54:53.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:53.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:54:53.451 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:53.449+0000 7f307ec160c0 -1 Falling back to public interface 2021-07-26T06:54:53.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:54.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:54.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:54.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:54:54.542 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:54:54.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:54.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:54:54.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:55.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:55.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:55.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:54:55.817 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:54:55.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:55.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:54:55.954 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:54:55.953+0000 7f307ec160c0 -1 osd.0 22 log_to_monitors {default=true} 2021-07-26T06:54:56.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:54:57.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:54:57.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:54:57.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:54:57.091 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:54:57.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:54:57.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:54:57.363 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/297623,v1:127.0.0.1:6803/297623] [v2:127.0.0.1:6804/297623,v1:127.0.0.1:6805/297623] exists,up 496c3c79-e98d-4917-9c9c-bb9178d538c5 2021-07-26T06:54:57.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:54:57.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:54:57.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:54:57.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:54:57.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:54:57.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:54:57.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:54:57.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:54:57.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:54:57.366 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:54:57.366 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:54:57.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:54:57.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:54:57.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:54:57.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:54:57.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:54:57.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:54:57.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:54:57.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:54:57.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:54:57.749 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T06:54:57.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:54:57.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:57.750 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:54:57.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=111669149698 2021-07-26T06:54:57.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 111669149698 2021-07-26T06:54:57.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149698' 2021-07-26T06:54:57.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:54:57.868 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:54:57.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672966 2021-07-26T06:54:57.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672966 2021-07-26T06:54:57.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149698 1-42949672966' 2021-07-26T06:54:57.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:57.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-111669149698 2021-07-26T06:54:57.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:57.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:54:57.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-111669149698 2021-07-26T06:54:57.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:57.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=111669149698 2021-07-26T06:54:57.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2021-07-26T06:54:57.991 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 111669149698 2021-07-26T06:54:57.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:54:58.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 111669149698 2021-07-26T06:54:58.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:54:59.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:54:59.257 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:54:59.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149698 -lt 111669149698 2021-07-26T06:54:59.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:54:59.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672966 2021-07-26T06:54:59.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:54:59.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:54:59.543 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672966 2021-07-26T06:54:59.544 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:54:59.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672966 2021-07-26T06:54:59.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-07-26T06:54:59.545 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672966 2021-07-26T06:54:59.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:54:59.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672966 -lt 42949672966 2021-07-26T06:54:59.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:54:59.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:54:59.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:55:00.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:55:00.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:55:00.210 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:55:00.210 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:55:00.210 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:55:00.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:55:00.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:55:00.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:55:00.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:55:00.512 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:55:00.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:55:00.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:55:00.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:55:00.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:55:00.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:55:00.838 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg rbd SOMETHING 2021-07-26T06:55:00.839 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=rbd 2021-07-26T06:55:00.839 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:55:00.839 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map rbd SOMETHING 2021-07-26T06:55:00.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:55:01.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=1.3 2021-07-26T06:55:01.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 1.3 2021-07-26T06:55:01.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=1.3 2021-07-26T06:55:01.144 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 1.3 2021-07-26T06:55:01.144 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:55:01.144 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:55:01.145 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:55:01.145 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:55:01.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:54:42.810305+0000 2021-07-26T06:55:01.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 1.3 2021-07-26T06:55:01.724 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 1.3 on osd.1 to repair 2021-07-26T06:55:01.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 1.3 2021-07-26T06:54:42.810305+0000 2021-07-26T06:55:01.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.3 2021-07-26T06:55:01.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:54:42.810305+0000 2021-07-26T06:55:01.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:55:01.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:55:01.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:55:01.735 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:55:01.736 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:55:01.736 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:55:01.736 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:55:01.736 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:55:01.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:54:42.810305+0000 '>' 2021-07-26T06:54:42.810305+0000 2021-07-26T06:55:01.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:55:02.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:55:02.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:55:02.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:55:02.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:55:02.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:55:02.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:55:02.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:55:03.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:54:42.810305+0000 '>' 2021-07-26T06:54:42.810305+0000 2021-07-26T06:55:03.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:55:04.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:55:04.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:55:04.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:55:04.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:55:04.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:55:04.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:55:04.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:55:04.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:54:42.810305+0000 '>' 2021-07-26T06:54:42.810305+0000 2021-07-26T06:55:04.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:55:05.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:55:05.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:55:05.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:55:05.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:55:05.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:55:05.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:55:05.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:55:05.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:54:42.810305+0000 '>' 2021-07-26T06:54:42.810305+0000 2021-07-26T06:55:05.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:55:06.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:55:06.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:55:06.846 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:55:06.846 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:55:06.846 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:55:06.846 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:55:06.847 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:55:07.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:54:42.810305+0000 '>' 2021-07-26T06:54:42.810305+0000 2021-07-26T06:55:07.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:55:08.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:55:08.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:55:08.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:55:08.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:55:08.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:55:08.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:55:08.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:55:08.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:55:05.557161+0000 '>' 2021-07-26T06:54:42.810305+0000 2021-07-26T06:55:08.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:55:08.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:55:08.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:55:08.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:55:08.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:55:08.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:55:08.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:55:08.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:55:08.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:55:08.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:55:08.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:55:08.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:55:08.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:55:08.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:55:08.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:55:08.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:55:08.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:55:08.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:55:08.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:55:08.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:55:08.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:55:08.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:55:08.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:55:08.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:55:08.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2021-07-26T06:55:10.066 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2021-07-26T06:55:10.066 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:55:10.066 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:55:10.067 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:55:10.067 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:55:10.067 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:55:10.067 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:55:10.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:55:10.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:55:10.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:55:10.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:55:10.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:55:10.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:55:10.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:55:10.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:55:10.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:55:10.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:55:10.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:55:10.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:55:10.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:55:10.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:55:10.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:55:10.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:55:10.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:55:10.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:55:10.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:55:10.344 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:55:10.345 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:55:10.345 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:55:10.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:55:10.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:55:10.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:55:10.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:55:10.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:55:10.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:55:10.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:55:10.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:55:10.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:55:10.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:55:10.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:55:10.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:55:10.348 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:55:10.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:55:10.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:55:10.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:55:10.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:55:10.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:55:10.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:55:10.363 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:10.362+0000 7f5c503040c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:55:10.365 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:10.364+0000 7f5c503040c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:55:10.366 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:10.365+0000 7f5c503040c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:55:10.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:55:10.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:55:10.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:55:10.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:55:10.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:55:10.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:55:10.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:55:10.622 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:55:10.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:55:10.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:55:10.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:55:10.888 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:10.887+0000 7f5c503040c0 -1 Falling back to public interface 2021-07-26T06:55:11.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:55:11.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:55:11.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:55:11.889 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:55:11.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:55:11.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:55:12.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:55:13.145 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:13.145+0000 7f5c503040c0 -1 osd.1 27 log_to_monitors {default=true} 2021-07-26T06:55:13.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:55:13.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:55:13.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:55:13.164 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:55:13.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:55:13.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:55:13.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:55:14.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:55:14.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:55:14.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:55:14.528 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:55:14.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:55:14.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:55:14.803 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 31 up_thru 31 down_at 28 last_clean_interval [10,27) [v2:127.0.0.1:6810/299147,v1:127.0.0.1:6811/299147] [v2:127.0.0.1:6812/299147,v1:127.0.0.1:6813/299147] exists,up d81b7e7c-7669-4e02-a5b5-3a8031948699 2021-07-26T06:55:14.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:55:14.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:55:14.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:55:14.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:55:14.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:55:14.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:55:14.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:55:14.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:55:14.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:55:14.806 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:55:14.806 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:55:14.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:55:14.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:55:14.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:55:14.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:55:14.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:55:14.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:55:14.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:55:14.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:55:15.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:55:15.167 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T06:55:15.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:55:15.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:55:15.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:55:15.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=111669149702 2021-07-26T06:55:15.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 111669149702 2021-07-26T06:55:15.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149702' 2021-07-26T06:55:15.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:55:15.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:55:15.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=133143986178 2021-07-26T06:55:15.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 133143986178 2021-07-26T06:55:15.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149702 1-133143986178' 2021-07-26T06:55:15.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:55:15.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-111669149702 2021-07-26T06:55:15.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:55:15.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:55:15.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-111669149702 2021-07-26T06:55:15.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:55:15.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=111669149702 2021-07-26T06:55:15.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2021-07-26T06:55:15.475 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 111669149702 2021-07-26T06:55:15.475 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:55:15.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149701 -lt 111669149702 2021-07-26T06:55:15.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:55:16.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:55:16.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:55:17.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149701 -lt 111669149702 2021-07-26T06:55:17.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:55:18.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:55:18.018 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:55:18.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149703 -lt 111669149702 2021-07-26T06:55:18.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:55:18.288 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-133143986178 2021-07-26T06:55:18.288 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:55:18.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:55:18.289 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-133143986178 2021-07-26T06:55:18.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:55:18.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=133143986178 2021-07-26T06:55:18.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 133143986178' 2021-07-26T06:55:18.291 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 133143986178 2021-07-26T06:55:18.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:55:18.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 133143986178 -lt 133143986178 2021-07-26T06:55:18.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:55:18.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:55:18.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:55:18.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:55:18.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:55:18.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:55:18.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:55:18.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:55:18.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:55:18.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:55:18.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:55:19.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:55:19.228 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:55:19.229 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:55:19.229 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:55:19.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:55:19.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:55:19.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:55:19.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:55:25.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:55:25.172 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:106: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2021-07-26T06:55:25.172 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=rbd 2021-07-26T06:55:25.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T06:55:25.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map rbd SOMETHING 2021-07-26T06:55:25.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:55:25.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:106: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2021-07-26T06:55:25.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2021-07-26T06:55:25.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=rbd 2021-07-26T06:55:25.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=1 2021-07-26T06:55:25.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:55:25.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:55:25.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:55:25.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:55:25.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:55:25.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:55:25.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:55:25.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:55:25.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:55:25.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:55:25.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:55:25.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:55:25.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:55:25.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:55:25.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:55:25.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:55:25.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:55:25.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2021-07-26T06:55:25.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:55:25.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:55:25.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:55:25.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:55:25.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:55:25.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2021-07-26T06:55:26.935 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:55:26.935 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:55:26.935 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:55:26.935 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:55:26.936 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:eb822e21:::SOMETHING:head# 2021-07-26T06:55:27.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:55:27.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:55:27.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:55:27.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:55:27.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:55:27.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:55:27.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:55:27.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:55:27.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:55:27.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:55:27.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:55:27.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:55:27.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:55:27.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:55:27.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:55:27.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:55:27.466 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:55:27.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:55:27.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:55:27.468 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:55:27.468 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:55:27.468 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:55:27.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:55:27.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:55:27.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:55:27.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:55:27.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:55:27.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:55:27.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:55:27.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:55:27.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:55:27.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:55:27.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:55:27.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:55:27.474 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:55:27.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:55:27.475 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:55:27.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:55:27.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:55:27.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:55:27.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:55:27.478 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:27.478+0000 7fdba77f20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:55:27.480 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:27.480+0000 7fdba77f20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:55:27.481 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:27.481+0000 7fdba77f20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:55:27.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:55:27.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:55:27.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:55:27.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:55:27.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:55:27.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:55:27.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:55:27.742 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:55:27.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:55:27.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:55:28.004 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:28.004+0000 7fdba77f20c0 -1 Falling back to public interface 2021-07-26T06:55:28.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:55:29.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:55:29.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:55:29.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:55:29.032 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:55:29.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:55:29.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:55:29.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:55:30.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:55:30.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:55:30.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:55:30.312 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:55:30.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:55:30.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:55:30.393 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:30.392+0000 7fdba77f20c0 -1 osd.1 32 log_to_monitors {default=true} 2021-07-26T06:55:30.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:55:31.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:55:31.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:55:31.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:55:31.596 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:55:31.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:55:31.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:55:31.893 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 36 up_thru 36 down_at 33 last_clean_interval [31,32) [v2:127.0.0.1:6810/300521,v1:127.0.0.1:6811/300521] [v2:127.0.0.1:6812/300521,v1:127.0.0.1:6813/300521] exists,up d81b7e7c-7669-4e02-a5b5-3a8031948699 2021-07-26T06:55:31.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:55:31.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:55:31.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:55:31.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:55:31.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:55:31.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:55:31.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:55:31.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:55:31.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:55:31.896 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:55:31.896 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:55:31.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:55:31.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:55:31.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:55:31.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:55:31.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:55:31.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:55:31.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:55:31.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:55:32.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:55:32.253 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T06:55:32.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:55:32.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:55:32.254 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:55:32.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=111669149707 2021-07-26T06:55:32.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 111669149707 2021-07-26T06:55:32.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149707' 2021-07-26T06:55:32.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:55:32.373 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:55:32.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=154618822658 2021-07-26T06:55:32.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 154618822658 2021-07-26T06:55:32.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149707 1-154618822658' 2021-07-26T06:55:32.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:55:32.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-111669149707 2021-07-26T06:55:32.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:55:32.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:55:32.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-111669149707 2021-07-26T06:55:32.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:55:32.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=111669149707 2021-07-26T06:55:32.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 111669149707' 2021-07-26T06:55:32.497 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 111669149707 2021-07-26T06:55:32.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:55:32.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149706 -lt 111669149707 2021-07-26T06:55:32.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:55:33.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:55:33.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:55:34.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149707 -lt 111669149707 2021-07-26T06:55:34.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:55:34.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-154618822658 2021-07-26T06:55:34.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:55:34.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:55:34.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-154618822658 2021-07-26T06:55:34.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:55:34.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=154618822658 2021-07-26T06:55:34.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 154618822658' 2021-07-26T06:55:34.040 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 154618822658 2021-07-26T06:55:34.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:55:34.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 154618822658 -lt 154618822658 2021-07-26T06:55:34.306 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:55:34.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:55:34.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:55:34.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:55:34.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:55:34.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:55:34.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:55:34.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:55:34.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:55:34.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:55:34.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:55:34.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:55:34.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:55:34.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:55:34.965 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:55:35.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:55:35.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:55:35.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:55:35.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg rbd SOMETHING 2021-07-26T06:55:35.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=rbd 2021-07-26T06:55:35.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T06:55:35.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map rbd SOMETHING 2021-07-26T06:55:35.292 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:55:35.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=1.3 2021-07-26T06:55:35.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 1.3 2021-07-26T06:55:35.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=1.3 2021-07-26T06:55:35.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 1.3 2021-07-26T06:55:35.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:55:35.593 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:55:35.593 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:55:35.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:55:35.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T06:55:05.557161+0000 2021-07-26T06:55:35.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 1.3 2021-07-26T06:55:36.144 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 1.3 on osd.1 to repair 2021-07-26T06:55:36.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 1.3 2021-07-26T06:55:05.557161+0000 2021-07-26T06:55:36.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.3 2021-07-26T06:55:36.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:55:05.557161+0000 2021-07-26T06:55:36.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:55:36.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:55:36.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:55:36.156 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:55:36.156 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:55:36.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:55:36.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:55:36.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:55:36.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:55:05.557161+0000 '>' 2021-07-26T06:55:05.557161+0000 2021-07-26T06:55:36.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:55:37.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:55:37.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:55:37.417 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:55:37.417 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:55:37.417 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:55:37.418 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:55:37.418 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:55:37.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:55:05.557161+0000 '>' 2021-07-26T06:55:05.557161+0000 2021-07-26T06:55:37.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:55:38.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:55:38.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:55:38.695 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:55:38.695 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:55:38.695 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:55:38.695 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:55:38.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:55:38.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:55:05.557161+0000 '>' 2021-07-26T06:55:05.557161+0000 2021-07-26T06:55:38.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:55:39.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:55:39.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:55:39.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:55:39.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:55:39.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:55:39.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:55:39.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:55:40.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:55:05.557161+0000 '>' 2021-07-26T06:55:05.557161+0000 2021-07-26T06:55:40.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:55:41.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:55:41.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:55:41.242 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T06:55:41.242 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T06:55:41.242 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:55:41.243 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:55:41.243 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T06:55:41.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:55:37.541957+0000 '>' 2021-07-26T06:55:05.557161+0000 2021-07-26T06:55:41.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:55:41.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:55:41.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:55:41.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:55:41.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:55:41.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:55:41.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:55:41.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:55:41.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:55:41.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:55:41.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:55:41.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:55:41.517 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:55:41.518 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:55:41.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:55:41.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:55:41.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:55:41.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:55:41.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2021-07-26T06:55:41.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:55:41.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:55:41.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:55:41.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:55:41.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:55:41.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2021-07-26T06:55:42.949 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2021-07-26T06:55:42.949 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:55:42.950 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:55:42.950 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:55:42.950 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:55:42.951 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T06:55:42.951 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T06:55:43.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:55:43.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:55:43.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:55:43.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:55:43.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:55:43.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:55:43.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:55:43.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:55:43.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:55:43.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:55:43.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:55:43.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:55:43.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:55:43.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:55:43.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:55:43.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:55:43.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:55:43.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:55:43.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:55:43.474 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:55:43.474 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:55:43.474 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:55:43.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:55:43.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:55:43.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:55:43.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:55:43.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:55:43.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:55:43.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:55:43.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:55:43.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:55:43.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:55:43.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:55:43.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:55:43.477 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:55:43.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:55:43.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:55:43.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:55:43.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:55:43.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:55:43.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:55:43.494 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:43.493+0000 7fd4363760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:55:43.495 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:43.495+0000 7fd4363760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:55:43.497 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:43.496+0000 7fd4363760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:55:43.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:55:43.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:55:43.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:55:43.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:55:43.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:55:43.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:55:43.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:55:43.751 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:55:43.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:55:43.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:55:44.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:55:44.020 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:44.020+0000 7fd4363760c0 -1 Falling back to public interface 2021-07-26T06:55:45.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:55:45.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:55:45.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:55:45.019 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:55:45.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:55:45.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:55:45.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:55:46.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:55:46.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:55:46.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:55:46.323 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:55:46.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:55:46.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:55:46.354 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:55:46.354+0000 7fd4363760c0 -1 osd.1 37 log_to_monitors {default=true} 2021-07-26T06:55:46.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:55:47.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:55:47.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:55:47.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:55:47.645 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:55:47.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:55:47.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:55:47.992 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [36,37) [v2:127.0.0.1:6810/302019,v1:127.0.0.1:6811/302019] [v2:127.0.0.1:6812/302019,v1:127.0.0.1:6813/302019] exists,up d81b7e7c-7669-4e02-a5b5-3a8031948699 2021-07-26T06:55:47.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:55:47.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:55:47.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:55:47.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:55:47.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:55:47.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:55:47.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:55:47.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:55:47.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:55:47.995 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:55:47.995 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:55:47.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:55:47.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:55:47.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:55:48.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:55:48.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:55:48.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:55:48.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:55:48.088 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:55:48.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:55:48.354 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T06:55:48.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:55:48.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:55:48.354 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:55:48.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=111669149711 2021-07-26T06:55:48.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 111669149711 2021-07-26T06:55:48.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149711' 2021-07-26T06:55:48.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:55:48.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:55:48.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=176093659138 2021-07-26T06:55:48.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 176093659138 2021-07-26T06:55:48.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149711 1-176093659138' 2021-07-26T06:55:48.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:55:48.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-111669149711 2021-07-26T06:55:48.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:55:48.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:55:48.593 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-111669149711 2021-07-26T06:55:48.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:55:48.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=111669149711 2021-07-26T06:55:48.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 111669149711' 2021-07-26T06:55:48.594 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 111669149711 2021-07-26T06:55:48.595 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:55:48.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149710 -lt 111669149711 2021-07-26T06:55:48.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:55:49.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:55:49.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:55:50.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149711 -lt 111669149711 2021-07-26T06:55:50.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:55:50.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-176093659138 2021-07-26T06:55:50.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:55:50.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:55:50.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-176093659138 2021-07-26T06:55:50.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:55:50.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=176093659138 2021-07-26T06:55:50.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2021-07-26T06:55:50.136 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 176093659138 2021-07-26T06:55:50.136 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:55:50.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 176093659138 -lt 176093659138 2021-07-26T06:55:50.402 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:55:50.402 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:55:50.403 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:55:50.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T06:55:50.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:55:50.730 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:55:50.730 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:55:50.730 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:55:50.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:55:50.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:55:50.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:55:50.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T06:55:50.992 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:55:50.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:55:50.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:55:51.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T06:55:51.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:55:51.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:55:51.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T06:55:58.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T06:55:58.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:55:58.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:55:58.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:55:58.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:55:58.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:55:58.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:55:58.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:55:58.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:55:58.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:55:58.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:55:58.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:55:58.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:55:58.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:55:58.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:55:58.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:55:58.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:55:58.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:55:58.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:55:58.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:55:58.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:55:58.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:55:58.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:55:58.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:55:58.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:55:58.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:55:58.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:55:58.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:55:58.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:55:58.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:55:58.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:55:58.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:55:58.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:55:58.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:55:58.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:55:58.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:55:58.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:55:58.461 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:55:58.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:55:58.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:55:58.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:55:58.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:55:58.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:55:58.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:55:58.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:55:58.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:55:58.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:55:58.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:55:58.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:55:58.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:55:58.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:55:58.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:55:58.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:55:58.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:55:58.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:55:58.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:55:58.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:55:58.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:55:58.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:55:58.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:55:58.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:55:58.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:55:58.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:55:58.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:55:58.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:55:58.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:55:58.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:55:58.475 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:55:58.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:55:58.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:55:58.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:55:58.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2021-07-26T06:55:58.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2021-07-26T06:55:58.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2021-07-26T06:55:58.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: local allow_overwrites=false 2021-07-26T06:55:58.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3611: corrupt_scrub_erasure: local poolname=ecpool 2021-07-26T06:55:58.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: local total_objs=7 2021-07-26T06:55:58.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2021-07-26T06:55:58.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:55:58.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:55:58.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:55:58.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:55:58.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:55:58.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T06:55:58.838 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:55:58.838 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:55:58.839 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:55:58.839 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:55:58.839 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:55:58.839 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:55:58.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:55:58.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:55:59.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:55:59.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:55:59.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:55:59.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:55:59.100 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:55:59.100 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:55:59.100 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:55:59.100 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:55:59.101 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:55:59.102 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:55:59.102 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:55:59.102 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:55:59.102 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:55:59.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:55:59.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:55:59.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:55:59.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:55:59.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:55:59.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:55:59.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:55:59.202 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:55:59.203 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:55:59.203 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:55:59.203 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:55:59.203 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:55:59.204 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:55:59.204 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:55:59.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:55:59.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:55:59.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3615: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2021-07-26T06:55:59.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:55:59.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:55:59.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:55:59.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:55:59.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:55:59.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:56:00.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:56:00.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:56:00.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:56:00.118 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:56:00.119 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:56:00.119 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:56:00.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:56:00.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:56:00.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:56:00.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: seq 0 2 2021-07-26T06:56:00.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: for id in $(seq 0 2) 2021-07-26T06:56:00.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3617: corrupt_scrub_erasure: '[' false = true ']' 2021-07-26T06:56:00.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3620: corrupt_scrub_erasure: run_osd_filestore td/osd-scrub-repair 0 2021-07-26T06:56:00.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:56:00.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:56:00.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2021-07-26T06:56:00.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:56:00.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:56:00.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:56:00.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:56:00.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:56:00.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:56:00.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:56:00.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:56:00.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:56:00.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:56:00.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:56:00.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:56:00.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:56:00.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:56:00.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:56:00.149 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:56:00.150 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:56:00.150 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:56:00.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:56:00.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:56:00.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:56:00.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:56:00.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:56:00.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:56:00.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:56:00.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:56:00.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:56:00.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:56:00.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:56:00.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:56:00.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:56:00.153 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:56:00.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=02a6c4ed-978b-40d7-8fc6-dfa7d25e4130 2021-07-26T06:56:00.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd1 02a6c4ed-978b-40d7-8fc6-dfa7d25e4130' 2021-07-26T06:56:00.154 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 02a6c4ed-978b-40d7-8fc6-dfa7d25e4130 2021-07-26T06:56:00.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:56:00.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQCAXP5gukV7CRAAnQO0AJLBOzocY3EY6H90Wg== 2021-07-26T06:56:00.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQCAXP5gukV7CRAAnQO0AJLBOzocY3EY6H90Wg=="}' 2021-07-26T06:56:00.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 02a6c4ed-978b-40d7-8fc6-dfa7d25e4130 -i td/osd-scrub-repair/0/new.json 2021-07-26T06:56:00.456 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:56:00.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:56:00.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCAXP5gukV7CRAAnQO0AJLBOzocY3EY6H90Wg== --osd-uuid 02a6c4ed-978b-40d7-8fc6-dfa7d25e4130 --osd-objectstore=filestore 2021-07-26T06:56:00.492 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:00.491+0000 7ffb829680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:00.493 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:00.492+0000 7ffb829680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:00.495 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:00.493+0000 7ffb829680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:00.967 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:00.966+0000 7ffb829680c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:56:01.429 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:01.427+0000 7ffb829680c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:56:01.429 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:01.427+0000 7ffb829680c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:56:01.429 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:01.427+0000 7ffb829680c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:56:01.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:56:01.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:56:01.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd0 key to auth repository 2021-07-26T06:56:01.922 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:56:01.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:56:02.094 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2021-07-26T06:56:02.095 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:56:02.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.0 2021-07-26T06:56:02.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:56:02.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:56:02.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:56:02.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:56:02.118 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:02.117+0000 7f4db11090c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:02.120 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:02.119+0000 7f4db11090c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:02.121 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:02.120+0000 7f4db11090c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:02.122 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:02.121+0000 7f4db11090c0 -1 Falling back to public interface 2021-07-26T06:56:02.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 0 2021-07-26T06:56:02.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:56:02.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:56:02.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:56:02.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:56:02.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:56:02.430 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:56:02.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:56:02.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:56:02.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:56:02.635 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:02.633+0000 7f4db11090c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:56:02.635 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:02.634+0000 7f4db11090c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:56:02.635 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:02.634+0000 7f4db11090c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:56:02.643 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:02.642+0000 7f4db11090c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:56:02.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:56:03.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:56:03.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:56:03.684 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:56:03.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:56:03.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:56:03.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:56:03.859 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/303268,v1:127.0.0.1:6801/303268] [v2:127.0.0.1:6802/303268,v1:127.0.0.1:6803/303268] exists,up 02a6c4ed-978b-40d7-8fc6-dfa7d25e4130 2021-07-26T06:56:03.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:56:03.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:56:03.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:56:03.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: for id in $(seq 0 2) 2021-07-26T06:56:03.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3617: corrupt_scrub_erasure: '[' false = true ']' 2021-07-26T06:56:03.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3620: corrupt_scrub_erasure: run_osd_filestore td/osd-scrub-repair 1 2021-07-26T06:56:03.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:56:03.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:56:03.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2021-07-26T06:56:03.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:56:03.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:56:03.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:56:03.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:56:03.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:56:03.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:56:03.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:56:03.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:56:03.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:56:03.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:56:03.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:56:03.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:56:03.868 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:56:03.868 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:56:03.869 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:56:03.869 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:56:03.869 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:56:03.869 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:56:03.869 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:56:03.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:56:03.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:56:03.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:56:03.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:56:03.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:56:03.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:56:03.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:56:03.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:56:03.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:56:03.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:56:03.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:56:03.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:56:03.873 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:56:03.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=bcf1ac15-08fb-47c5-9d33-a053d9f78c45 2021-07-26T06:56:03.873 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 bcf1ac15-08fb-47c5-9d33-a053d9f78c45 2021-07-26T06:56:03.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd1 bcf1ac15-08fb-47c5-9d33-a053d9f78c45' 2021-07-26T06:56:03.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:56:03.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQCDXP5gZl1HNBAAHfgOexUMYGvzc6OxKfrKXA== 2021-07-26T06:56:03.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQCDXP5gZl1HNBAAHfgOexUMYGvzc6OxKfrKXA=="}' 2021-07-26T06:56:03.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new bcf1ac15-08fb-47c5-9d33-a053d9f78c45 -i td/osd-scrub-repair/1/new.json 2021-07-26T06:56:04.150 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:56:04.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:56:04.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCDXP5gZl1HNBAAHfgOexUMYGvzc6OxKfrKXA== --osd-uuid bcf1ac15-08fb-47c5-9d33-a053d9f78c45 --osd-objectstore=filestore 2021-07-26T06:56:04.186 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:04.185+0000 7fd6e13c00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:04.200 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:04.199+0000 7fd6e13c00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:04.201 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:04.200+0000 7fd6e13c00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:04.695 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:04.693+0000 7fd6e13c00c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:56:05.566 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:05.565+0000 7fd6e13c00c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:56:05.575 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:05.573+0000 7fd6e13c00c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:56:05.575 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:05.573+0000 7fd6e13c00c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:56:06.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:56:06.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:56:06.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd1 key to auth repository 2021-07-26T06:56:06.300 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:56:06.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:56:06.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.1 2021-07-26T06:56:06.636 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:56:06.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:56:06.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:56:06.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:56:06.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:56:06.658 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:06.657+0000 7fe43118b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:06.660 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:06.659+0000 7fe43118b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:06.661 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:06.660+0000 7fe43118b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:06.662 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:06.661+0000 7fe43118b0c0 -1 Falling back to public interface 2021-07-26T06:56:07.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 1 2021-07-26T06:56:07.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:56:07.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:56:07.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:56:07.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:56:07.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:56:07.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:56:07.038 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:56:07.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:56:07.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:56:07.233 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:07.231+0000 7fe43118b0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:56:07.242 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:07.240+0000 7fe43118b0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:56:07.242 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:07.240+0000 7fe43118b0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:56:07.248 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:07.247+0000 7fe43118b0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:56:07.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:56:08.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:56:08.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:56:08.322 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:56:08.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:56:08.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:56:08.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:56:08.592 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/303641,v1:127.0.0.1:6811/303641] [v2:127.0.0.1:6812/303641,v1:127.0.0.1:6813/303641] exists,up bcf1ac15-08fb-47c5-9d33-a053d9f78c45 2021-07-26T06:56:08.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:56:08.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:56:08.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:56:08.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: for id in $(seq 0 2) 2021-07-26T06:56:08.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3617: corrupt_scrub_erasure: '[' false = true ']' 2021-07-26T06:56:08.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3620: corrupt_scrub_erasure: run_osd_filestore td/osd-scrub-repair 2 2021-07-26T06:56:08.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T06:56:08.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:56:08.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2021-07-26T06:56:08.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:56:08.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:56:08.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:56:08.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:56:08.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:56:08.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:56:08.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:56:08.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:56:08.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:56:08.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:56:08.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:56:08.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:56:08.603 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:56:08.603 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:56:08.603 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:56:08.604 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:56:08.604 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:56:08.604 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:56:08.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:56:08.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:56:08.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:56:08.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:56:08.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:56:08.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:56:08.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:56:08.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:56:08.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:56:08.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:56:08.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:56:08.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:56:08.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:56:08.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:56:08.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=8d19ee35-ef93-4bea-8512-476d49af717d 2021-07-26T06:56:08.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd1 8d19ee35-ef93-4bea-8512-476d49af717d' 2021-07-26T06:56:08.608 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 8d19ee35-ef93-4bea-8512-476d49af717d 2021-07-26T06:56:08.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:56:08.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQCIXP5gUfxtJBAAjhIv2hojtXw474IIjNgeCg== 2021-07-26T06:56:08.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQCIXP5gUfxtJBAAjhIv2hojtXw474IIjNgeCg=="}' 2021-07-26T06:56:08.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 8d19ee35-ef93-4bea-8512-476d49af717d -i td/osd-scrub-repair/2/new.json 2021-07-26T06:56:08.967 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:56:08.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2021-07-26T06:56:08.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCIXP5gUfxtJBAAjhIv2hojtXw474IIjNgeCg== --osd-uuid 8d19ee35-ef93-4bea-8512-476d49af717d --osd-objectstore=filestore 2021-07-26T06:56:09.004 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:09.003+0000 7f16667920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:09.016 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:09.015+0000 7f16667920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:09.017 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:09.016+0000 7f16667920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:09.647 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:09.646+0000 7f16667920c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:56:10.134 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:10.133+0000 7f16667920c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:56:10.134 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:10.133+0000 7f16667920c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:56:10.134 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:10.133+0000 7f16667920c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:56:10.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T06:56:10.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:56:10.724 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:56:10.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd2 key to auth repository 2021-07-26T06:56:10.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:56:11.081 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2021-07-26T06:56:11.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.2 2021-07-26T06:56:11.083 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:56:11.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:56:11.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:56:11.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:56:11.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:56:11.105 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:11.105+0000 7fd44138e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:11.107 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:11.107+0000 7fd44138e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:11.108 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:11.108+0000 7fd44138e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:11.109 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:11.109+0000 7fd44138e0c0 -1 Falling back to public interface 2021-07-26T06:56:11.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 2 2021-07-26T06:56:11.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:56:11.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:56:11.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:56:11.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:56:11.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:56:11.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:56:11.496 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:56:11.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:56:11.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:56:11.617 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:11.616+0000 7fd44138e0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:56:11.632 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:11.632+0000 7fd44138e0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:56:11.632 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:11.632+0000 7fd44138e0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:56:11.638 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:11.638+0000 7fd44138e0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:56:11.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:56:12.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:56:12.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:56:12.781 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:56:12.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:56:12.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:56:12.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:56:13.127 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/303924,v1:127.0.0.1:6819/303924] [v2:127.0.0.1:6820/303924,v1:127.0.0.1:6821/303924] exists,up 8d19ee35-ef93-4bea-8512-476d49af717d 2021-07-26T06:56:13.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:56:13.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:56:13.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:56:13.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: create_rbd_pool 2021-07-26T06:56:13.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:56:13.386 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T06:56:13.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:56:13.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:56:13.829 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T06:56:13.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:56:14.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:56:16.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3624: corrupt_scrub_erasure: create_pool foo 1 2021-07-26T06:56:16.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2021-07-26T06:56:17.286 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'foo' created 2021-07-26T06:56:17.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:56:18.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3626: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2021-07-26T06:56:18.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=ecpool 2021-07-26T06:56:18.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T06:56:18.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=false 2021-07-26T06:56:18.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T06:56:18.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2021-07-26T06:56:18.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2021-07-26T06:56:18.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2021-07-26T06:56:19.465 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' created 2021-07-26T06:56:19.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:56:20.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' false = true ']' 2021-07-26T06:56:20.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T06:56:20.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:56:20.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:56:20.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:56:20.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:56:20.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:56:20.478 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:56:20.479 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:56:20.479 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:56:20.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:56:20.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:56:20.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:56:20.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:56:20.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:56:20.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:56:20.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:56:20.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:56:20.858 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:56:20.858 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:56:20.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:56:20.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:20.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:56:20.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836485 2021-07-26T06:56:20.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836485 2021-07-26T06:56:20.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836485' 2021-07-26T06:56:20.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:20.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:56:21.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672964 2021-07-26T06:56:21.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672964 2021-07-26T06:56:21.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836485 1-42949672964' 2021-07-26T06:56:21.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:21.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:56:21.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509443 2021-07-26T06:56:21.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509443 2021-07-26T06:56:21.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836485 1-42949672964 2-64424509443' 2021-07-26T06:56:21.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:21.214 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836485 2021-07-26T06:56:21.214 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:21.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:56:21.216 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836485 2021-07-26T06:56:21.216 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:21.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836485 2021-07-26T06:56:21.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2021-07-26T06:56:21.217 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836485 2021-07-26T06:56:21.217 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:21.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836484 -lt 21474836485 2021-07-26T06:56:21.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:56:22.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:56:22.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:22.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836484 -lt 21474836485 2021-07-26T06:56:22.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:56:23.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:56:23.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:24.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836485 -lt 21474836485 2021-07-26T06:56:24.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:24.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672964 2021-07-26T06:56:24.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:24.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:56:24.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672964 2021-07-26T06:56:24.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:24.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672964 2021-07-26T06:56:24.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2021-07-26T06:56:24.038 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672964 2021-07-26T06:56:24.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:56:24.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672964 -lt 42949672964 2021-07-26T06:56:24.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:24.362 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509443 2021-07-26T06:56:24.362 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:24.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:56:24.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509443 2021-07-26T06:56:24.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:24.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509443 2021-07-26T06:56:24.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2021-07-26T06:56:24.365 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509443 2021-07-26T06:56:24.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:56:24.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509443 -lt 64424509443 2021-07-26T06:56:24.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:56:24.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:56:24.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:56:24.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:56:24.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:56:24.980 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:56:24.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:56:24.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:56:24.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:56:24.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:56:24.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:56:25.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:56:25.252 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:56:25.252 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:56:25.252 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:56:25.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:56:25.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:56:25.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:56:25.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T06:56:25.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3627: corrupt_scrub_erasure: wait_for_clean 2021-07-26T06:56:25.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:56:25.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:56:25.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:56:25.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:56:25.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:56:25.580 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:56:25.581 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:56:25.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:56:25.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:56:25.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:56:25.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:56:25.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:56:25.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:56:25.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:56:25.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:56:25.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:56:25.940 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:56:25.940 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:56:25.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:56:25.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:25.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:56:26.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836487 2021-07-26T06:56:26.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836487 2021-07-26T06:56:26.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487' 2021-07-26T06:56:26.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:26.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:56:26.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672966 2021-07-26T06:56:26.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672966 2021-07-26T06:56:26.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2021-07-26T06:56:26.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:26.218 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:56:26.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509445 2021-07-26T06:56:26.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509445 2021-07-26T06:56:26.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672966 2-64424509445' 2021-07-26T06:56:26.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:26.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836487 2021-07-26T06:56:26.337 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:26.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:56:26.338 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836487 2021-07-26T06:56:26.338 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:26.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836487 2021-07-26T06:56:26.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-07-26T06:56:26.339 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836487 2021-07-26T06:56:26.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:26.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836485 -lt 21474836487 2021-07-26T06:56:26.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:56:27.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:56:27.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:27.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836487 2021-07-26T06:56:27.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:27.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672966 2021-07-26T06:56:27.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:27.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:56:27.873 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672966 2021-07-26T06:56:27.873 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:27.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672966 2021-07-26T06:56:27.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-07-26T06:56:27.875 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672966 2021-07-26T06:56:27.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:56:28.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672966 -lt 42949672966 2021-07-26T06:56:28.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:28.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509445 2021-07-26T06:56:28.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:28.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:56:28.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509445 2021-07-26T06:56:28.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:28.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509445 2021-07-26T06:56:28.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2021-07-26T06:56:28.188 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509445 2021-07-26T06:56:28.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:56:28.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509445 -lt 64424509445 2021-07-26T06:56:28.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:56:28.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:56:28.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:56:28.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:56:28.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:56:28.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:56:28.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:56:28.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:56:28.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:56:28.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:56:28.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:56:29.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:56:29.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:56:29.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:56:29.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:56:29.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:56:29.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:56:29.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:56:29.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: seq 1 7 2021-07-26T06:56:29.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T06:56:29.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ1 2021-07-26T06:56:29.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2021-07-26T06:56:29.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:56:29.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T06:56:29.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ1 2021-07-26T06:56:29.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:56:29.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:56:29.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:56:29.991 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:56:30.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:56:30.401 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:56:30.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:56:30.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:56:30.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:56:30.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 1 % 2 2021-07-26T06:56:30.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=1 2021-07-26T06:56:30.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T06:56:30.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: local payload=UVWXYZZZ 2021-07-26T06:56:30.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: echo UVWXYZZZ 2021-07-26T06:56:30.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3640: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:56:30.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:56:30.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:56:30.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:56:30.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:56:30.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:56:30.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:56:30.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:56:30.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:56:30.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:56:30.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:56:30.549 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:56:30.549 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:56:30.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:56:30.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:56:30.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:56:30.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:56:30.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:56:30.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:56:30.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:56:30.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:56:30.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:56:30.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:56:30.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:56:31.533 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:56:31.534 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T06:56:31.534 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:56:31.534 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s0_head,0#3:00000000::::head#, (61) No data available 2021-07-26T06:56:31.534 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:56:31.535 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:56:31.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:56:31.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:56:31.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:56:31.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:56:31.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:56:31.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:56:31.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:56:31.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:56:31.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:56:31.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:56:31.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:56:31.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:56:31.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:56:31.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:56:31.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:56:31.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:56:31.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:56:31.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:56:31.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:56:31.989 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:56:31.990 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:56:31.990 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:56:31.990 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:56:31.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:56:31.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:56:31.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:56:31.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:56:31.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:56:31.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:56:31.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:56:31.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:56:31.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:56:31.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:56:31.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:56:31.993 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:56:31.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:56:31.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:56:31.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:56:31.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:56:31.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:56:31.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:56:32.007 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:32.007+0000 7fd90d9ed0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:32.009 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:32.009+0000 7fd90d9ed0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:32.010 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:32.010+0000 7fd90d9ed0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:32.011 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:32.011+0000 7fd90d9ed0c0 -1 Falling back to public interface 2021-07-26T06:56:32.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:56:32.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:56:32.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:56:32.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:56:32.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:56:32.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:56:32.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:56:32.266 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:56:32.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:56:32.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:56:32.416 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:32.415+0000 7fd90d9ed0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:56:32.428 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:32.427+0000 7fd90d9ed0c0 -1 journal do_read_entry(376832): bad header magic 2021-07-26T06:56:32.428 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:32.427+0000 7fd90d9ed0c0 -1 journal do_read_entry(376832): bad header magic 2021-07-26T06:56:32.435 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:32.435+0000 7fd90d9ed0c0 -1 osd.1 34 log_to_monitors {default=true} 2021-07-26T06:56:32.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:56:33.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:56:33.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:56:33.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:56:33.584 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:56:33.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:56:33.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:56:33.999 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/305150,v1:127.0.0.1:6811/305150] [v2:127.0.0.1:6812/305150,v1:127.0.0.1:6813/305150] exists,up bcf1ac15-08fb-47c5-9d33-a053d9f78c45 2021-07-26T06:56:34.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:56:34.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:56:34.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:56:34.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:56:34.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:56:34.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:56:34.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:56:34.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:56:34.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:56:34.002 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:56:34.002 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:56:34.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:56:34.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:56:34.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:56:34.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:56:34.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:56:34.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:56:34.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:56:34.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:56:34.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:56:34.383 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:56:34.384 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:56:34.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:56:34.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:34.384 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:56:34.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836489 2021-07-26T06:56:34.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836489 2021-07-26T06:56:34.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489' 2021-07-26T06:56:34.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:34.526 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:56:34.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=163208757250 2021-07-26T06:56:34.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 163208757250 2021-07-26T06:56:34.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-163208757250' 2021-07-26T06:56:34.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:34.643 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:56:34.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509448 2021-07-26T06:56:34.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509448 2021-07-26T06:56:34.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-163208757250 2-64424509448' 2021-07-26T06:56:34.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:34.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836489 2021-07-26T06:56:34.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:34.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:56:34.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836489 2021-07-26T06:56:34.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:34.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836489 2021-07-26T06:56:34.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2021-07-26T06:56:34.763 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836489 2021-07-26T06:56:34.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:35.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836489 2021-07-26T06:56:35.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:35.070 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-163208757250 2021-07-26T06:56:35.070 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:35.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:56:35.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-163208757250 2021-07-26T06:56:35.072 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:35.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=163208757250 2021-07-26T06:56:35.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2021-07-26T06:56:35.072 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 163208757250 2021-07-26T06:56:35.073 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:56:35.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 163208757250 -lt 163208757250 2021-07-26T06:56:35.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:35.384 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509448 2021-07-26T06:56:35.385 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:35.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:56:35.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509448 2021-07-26T06:56:35.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:35.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509448 2021-07-26T06:56:35.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2021-07-26T06:56:35.387 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509448 2021-07-26T06:56:35.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:56:35.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509448 -lt 64424509448 2021-07-26T06:56:35.660 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:56:35.660 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:56:35.660 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:56:36.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:56:36.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:56:36.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:56:36.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:56:36.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:56:36.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:56:36.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:56:36.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:56:36.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:56:36.321 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:56:36.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:56:36.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:56:36.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:56:36.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:56:36.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:56:36.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T06:56:36.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ2 2021-07-26T06:56:36.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2021-07-26T06:56:36.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:56:36.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T06:56:36.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ2 2021-07-26T06:56:36.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:56:36.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:56:36.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:56:37.222 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:56:37.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:56:37.876 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:56:37.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:56:37.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:56:37.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:56:47.765 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 2 % 2 2021-07-26T06:56:47.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=0 2021-07-26T06:56:47.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T06:56:47.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2021-07-26T06:56:47.769 INFO:tasks.workunit.client.0.smithi071.stderr:1+0 records in 2021-07-26T06:56:47.769 INFO:tasks.workunit.client.0.smithi071.stderr:1+0 records out 2021-07-26T06:56:47.770 INFO:tasks.workunit.client.0.smithi071.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 8.2088e-05 s, 24.9 MB/s 2021-07-26T06:56:47.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:56:47.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:56:47.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:56:47.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:56:47.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:56:47.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:56:47.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:56:47.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:56:47.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:56:47.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:56:47.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:56:47.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:56:47.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:56:47.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:56:47.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:56:47.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:56:47.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:56:47.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:56:47.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:56:47.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:56:47.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:56:47.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:56:47.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:56:47.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:56:48.746 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:56:48.747 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T06:56:48.747 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s2_head,2#3:00000000::::head#, (61) No data available 2021-07-26T06:56:48.747 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:56:48.747 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:56:48.748 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:56:49.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T06:56:49.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:56:49.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:56:49.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:56:49.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:56:49.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:56:49.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:56:49.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:56:49.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:56:49.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:56:49.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:56:49.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:56:49.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:56:49.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:56:49.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:56:49.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:56:49.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:56:49.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:56:49.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:56:49.244 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:56:49.244 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:56:49.245 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:56:49.245 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:56:49.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:56:49.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:56:49.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:56:49.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:56:49.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:56:49.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:56:49.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:56:49.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:56:49.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:56:49.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:56:49.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:56:49.247 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:56:49.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:56:49.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:56:49.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:56:49.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:56:49.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:56:49.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:56:49.263 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:49.262+0000 7fab703ec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:49.270 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:49.269+0000 7fab703ec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:49.271 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:49.270+0000 7fab703ec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:49.273 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:49.272+0000 7fab703ec0c0 -1 Falling back to public interface 2021-07-26T06:56:49.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:56:49.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:56:49.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:56:49.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:56:49.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:56:49.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:56:49.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:56:49.628 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:56:49.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:56:49.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:56:49.824 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:49.823+0000 7fab703ec0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:56:49.833 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:49.832+0000 7fab703ec0c0 -1 journal do_read_entry(647168): bad header magic 2021-07-26T06:56:49.833 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:49.832+0000 7fab703ec0c0 -1 journal do_read_entry(647168): bad header magic 2021-07-26T06:56:49.841 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:49.840+0000 7fab703ec0c0 -1 osd.0 42 log_to_monitors {default=true} 2021-07-26T06:56:49.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:56:50.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:56:50.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:56:50.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:56:50.905 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:56:50.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:56:50.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:56:51.357 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 46 up_thru 46 down_at 43 last_clean_interval [5,42) [v2:127.0.0.1:6800/305798,v1:127.0.0.1:6801/305798] [v2:127.0.0.1:6802/305798,v1:127.0.0.1:6803/305798] exists,up 02a6c4ed-978b-40d7-8fc6-dfa7d25e4130 2021-07-26T06:56:51.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:56:51.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:56:51.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:56:51.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:56:51.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:56:51.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:56:51.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:56:51.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:56:51.359 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:56:51.359 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:56:51.360 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:56:51.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:56:51.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:56:51.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:56:51.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:56:51.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:56:51.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:56:51.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:56:51.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:56:51.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:56:51.718 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:56:51.718 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:56:51.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:56:51.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:51.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:56:51.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=197568495618 2021-07-26T06:56:51.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 197568495618 2021-07-26T06:56:51.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-197568495618' 2021-07-26T06:56:51.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:51.880 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:56:51.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=163208757254 2021-07-26T06:56:51.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 163208757254 2021-07-26T06:56:51.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-197568495618 1-163208757254' 2021-07-26T06:56:51.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:51.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:56:52.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509452 2021-07-26T06:56:52.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509452 2021-07-26T06:56:52.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-197568495618 1-163208757254 2-64424509452' 2021-07-26T06:56:52.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:52.113 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-197568495618 2021-07-26T06:56:52.113 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:52.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:56:52.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-197568495618 2021-07-26T06:56:52.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:52.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=197568495618 2021-07-26T06:56:52.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 197568495618' 2021-07-26T06:56:52.116 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 197568495618 2021-07-26T06:56:52.116 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:52.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 197568495618 2021-07-26T06:56:52.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:56:53.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:56:53.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:53.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 197568495618 -lt 197568495618 2021-07-26T06:56:53.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:53.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-163208757254 2021-07-26T06:56:53.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:53.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:56:53.657 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-163208757254 2021-07-26T06:56:53.657 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:53.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=163208757254 2021-07-26T06:56:53.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 163208757254' 2021-07-26T06:56:53.659 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 163208757254 2021-07-26T06:56:53.659 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:56:53.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 163208757254 -lt 163208757254 2021-07-26T06:56:53.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:53.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509452 2021-07-26T06:56:53.926 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:53.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:56:53.926 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509452 2021-07-26T06:56:53.927 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:53.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509452 2021-07-26T06:56:53.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-07-26T06:56:53.928 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509452 2021-07-26T06:56:53.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:56:54.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509452 -lt 64424509452 2021-07-26T06:56:54.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:56:54.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:56:54.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:56:54.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:56:54.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:56:54.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:56:54.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:56:54.560 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:56:54.560 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:56:54.560 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:56:54.560 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:56:54.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:56:54.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:56:54.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:56:54.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:56:55.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:56:55.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:56:55.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:56:55.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T06:56:55.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ3 2021-07-26T06:56:55.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2021-07-26T06:56:55.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:56:55.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T06:56:55.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ3 2021-07-26T06:56:55.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:56:55.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:56:55.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:56:55.629 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:56:55.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:56:56.131 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:56:56.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:56:56.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:56:56.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:56:56.231 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 3 % 2 2021-07-26T06:56:56.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=1 2021-07-26T06:56:56.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T06:56:56.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3651: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2021-07-26T06:56:56.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:56:56.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:56:56.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:56:56.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:56:56.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2021-07-26T06:56:56.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:56:56.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:56:56.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:56:56.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:56:56.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:56:56.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:56:56.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:56:56.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:56:56.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:56:56.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:56:56.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:56:56.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2021-07-26T06:56:56.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:56:56.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:56:56.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:56:56.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:56:56.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:56:56.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2021-07-26T06:56:57.028 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:56:57.029 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T06:56:57.029 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:56:57.029 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s0_head,0#3:00000000::::head#, (61) No data available 2021-07-26T06:56:57.030 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:56:57.030 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:56:57.030 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2021-07-26T06:56:57.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:56:57.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:56:57.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:56:57.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:56:57.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:56:57.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:56:57.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:56:57.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:56:57.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:56:57.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:56:57.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:56:57.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:56:57.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:56:57.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:56:57.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:56:57.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:56:57.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:56:57.502 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:56:57.502 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:56:57.502 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:56:57.502 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:56:57.502 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:56:57.503 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:56:57.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:56:57.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:56:57.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:56:57.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:56:57.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:56:57.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:56:57.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:56:57.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:56:57.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:56:57.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:56:57.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:56:57.505 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:56:57.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:56:57.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:56:57.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:56:57.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:56:57.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:56:57.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:56:57.518 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:57.517+0000 7fa7307e40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:57.589 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:57.588+0000 7fa7307e40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:57.590 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:57.589+0000 7fa7307e40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:56:57.591 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:57.590+0000 7fa7307e40c0 -1 Falling back to public interface 2021-07-26T06:56:57.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:56:57.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:56:57.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:56:57.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:56:57.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:56:57.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:56:57.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:56:57.778 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:56:57.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:56:57.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:56:58.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:56:58.247 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:58.245+0000 7fa7307e40c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:56:58.259 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:58.257+0000 7fa7307e40c0 -1 journal do_read_entry(888832): bad header magic 2021-07-26T06:56:58.260 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:58.257+0000 7fa7307e40c0 -1 journal do_read_entry(888832): bad header magic 2021-07-26T06:56:58.266 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:56:58.264+0000 7fa7307e40c0 -1 osd.1 50 log_to_monitors {default=true} 2021-07-26T06:56:59.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:56:59.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:56:59.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:56:59.148 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:56:59.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:56:59.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:56:59.467 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 54 up_thru 54 down_at 51 last_clean_interval [38,50) [v2:127.0.0.1:6810/306472,v1:127.0.0.1:6811/306472] [v2:127.0.0.1:6812/306472,v1:127.0.0.1:6813/306472] exists,up bcf1ac15-08fb-47c5-9d33-a053d9f78c45 2021-07-26T06:56:59.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:56:59.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:56:59.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:56:59.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:56:59.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:56:59.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:56:59.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:56:59.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:56:59.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:56:59.469 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:56:59.470 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:56:59.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:56:59.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:56:59.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:56:59.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:56:59.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:56:59.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:56:59.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:56:59.564 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:56:59.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:56:59.870 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:56:59.870 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:56:59.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:56:59.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:59.871 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:57:00.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=197568495620 2021-07-26T06:57:00.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 197568495620 2021-07-26T06:57:00.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-197568495620' 2021-07-26T06:57:00.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:00.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:57:00.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=231928233986 2021-07-26T06:57:00.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 231928233986 2021-07-26T06:57:00.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-197568495620 1-231928233986' 2021-07-26T06:57:00.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:00.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:57:00.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509455 2021-07-26T06:57:00.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509455 2021-07-26T06:57:00.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-197568495620 1-231928233986 2-64424509455' 2021-07-26T06:57:00.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:00.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-197568495620 2021-07-26T06:57:00.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:00.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:57:00.286 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-197568495620 2021-07-26T06:57:00.286 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:00.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=197568495620 2021-07-26T06:57:00.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 197568495620' 2021-07-26T06:57:00.287 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 197568495620 2021-07-26T06:57:00.287 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:00.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 197568495619 -lt 197568495620 2021-07-26T06:57:00.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:57:01.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:57:01.551 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:01.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 197568495621 -lt 197568495620 2021-07-26T06:57:01.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:01.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-231928233986 2021-07-26T06:57:01.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:01.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:57:01.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-231928233986 2021-07-26T06:57:01.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:01.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=231928233986 2021-07-26T06:57:01.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 231928233986' 2021-07-26T06:57:01.831 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 231928233986 2021-07-26T06:57:01.831 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:57:02.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 231928233986 -lt 231928233986 2021-07-26T06:57:02.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:02.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509455 2021-07-26T06:57:02.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:02.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:57:02.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509455 2021-07-26T06:57:02.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:02.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509455 2021-07-26T06:57:02.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2021-07-26T06:57:02.135 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509455 2021-07-26T06:57:02.136 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:57:02.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509455 -lt 64424509455 2021-07-26T06:57:02.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:57:02.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:02.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:02.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:57:02.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:02.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:02.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:02.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:02.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:02.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:02.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:03.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:57:03.131 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:03.131 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:03.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:03.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:57:03.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:57:03.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:57:03.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T06:57:03.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ4 2021-07-26T06:57:03.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2021-07-26T06:57:03.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:57:03.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T06:57:03.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ4 2021-07-26T06:57:03.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:57:03.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:57:03.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:57:04.063 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:57:04.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:57:04.582 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:57:04.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:57:04.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:57:04.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:57:13.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 4 % 2 2021-07-26T06:57:13.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=0 2021-07-26T06:57:13.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T06:57:13.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3655: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2021-07-26T06:57:13.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3656: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2021-07-26T06:57:13.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: echo -n bad-val 2021-07-26T06:57:13.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3660: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2021-07-26T06:57:13.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:57:14.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:57:14.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:57:14.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:57:14.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2021-07-26T06:57:14.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:57:14.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:57:14.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:57:14.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:57:14.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:57:14.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:57:14.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:57:14.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:57:14.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:57:14.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:57:14.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:57:14.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2021-07-26T06:57:14.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:57:14.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:57:14.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:57:14.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:57:14.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:57:14.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2021-07-26T06:57:14.879 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:57:14.879 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T06:57:14.879 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s2_head,2#3:00000000::::head#, (61) No data available 2021-07-26T06:57:14.880 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:57:14.880 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:57:14.880 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:57:15.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T06:57:15.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:57:15.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:57:15.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:57:15.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:57:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:57:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:57:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:57:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:57:15.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:57:15.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:57:15.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:57:15.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:57:15.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:57:15.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:57:15.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:57:15.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:57:15.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:57:15.323 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:57:15.323 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:57:15.323 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:57:15.323 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:57:15.324 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:57:15.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:57:15.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:57:15.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:57:15.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:57:15.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:57:15.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:57:15.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:57:15.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:57:15.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:57:15.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:57:15.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:57:15.326 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:57:15.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:57:15.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:57:15.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:57:15.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:57:15.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:57:15.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:57:15.343 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:15.342+0000 7f69f0c040c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:15.363 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:15.363+0000 7f69f0c040c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:15.364 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:15.364+0000 7f69f0c040c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:15.366 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:15.366+0000 7f69f0c040c0 -1 Falling back to public interface 2021-07-26T06:57:15.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:57:15.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:57:15.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:57:15.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:57:15.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:57:15.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:57:15.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:57:15.728 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:57:15.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:57:15.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:57:15.879 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:15.878+0000 7f69f0c040c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:57:15.890 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:15.889+0000 7f69f0c040c0 -1 journal do_read_entry(1167360): bad header magic 2021-07-26T06:57:15.890 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:15.889+0000 7f69f0c040c0 -1 journal do_read_entry(1167360): bad header magic 2021-07-26T06:57:15.897 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:15.897+0000 7f69f0c040c0 -1 osd.0 58 log_to_monitors {default=true} 2021-07-26T06:57:15.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:57:16.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:57:16.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:57:16.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:57:16.998 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:57:16.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:57:16.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:57:17.276 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 62 up_thru 62 down_at 59 last_clean_interval [46,58) [v2:127.0.0.1:6800/307180,v1:127.0.0.1:6801/307180] [v2:127.0.0.1:6802/307180,v1:127.0.0.1:6803/307180] exists,up 02a6c4ed-978b-40d7-8fc6-dfa7d25e4130 2021-07-26T06:57:17.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:57:17.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:57:17.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:57:17.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:57:17.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:57:17.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:57:17.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:57:17.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:57:17.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:57:17.278 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:57:17.278 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:57:17.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:57:17.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:57:17.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:57:17.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:57:17.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:57:17.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:57:17.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:57:17.373 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:57:17.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:57:17.640 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:57:17.640 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:57:17.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:57:17.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:17.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:57:17.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=266287972354 2021-07-26T06:57:17.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 266287972354 2021-07-26T06:57:17.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-266287972354' 2021-07-26T06:57:17.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:17.795 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:57:17.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=231928233990 2021-07-26T06:57:17.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 231928233990 2021-07-26T06:57:17.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-266287972354 1-231928233990' 2021-07-26T06:57:17.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:17.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:57:18.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509459 2021-07-26T06:57:18.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509459 2021-07-26T06:57:18.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-266287972354 1-231928233990 2-64424509459' 2021-07-26T06:57:18.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:18.030 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-266287972354 2021-07-26T06:57:18.030 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:18.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:57:18.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-266287972354 2021-07-26T06:57:18.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:18.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=266287972354 2021-07-26T06:57:18.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 266287972354' 2021-07-26T06:57:18.033 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 266287972354 2021-07-26T06:57:18.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:18.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 266287972354 2021-07-26T06:57:18.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:57:19.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:57:19.296 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:19.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 266287972354 -lt 266287972354 2021-07-26T06:57:19.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:19.572 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-231928233990 2021-07-26T06:57:19.572 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:19.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:57:19.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-231928233990 2021-07-26T06:57:19.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:19.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=231928233990 2021-07-26T06:57:19.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 231928233990' 2021-07-26T06:57:19.574 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 231928233990 2021-07-26T06:57:19.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:57:19.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 231928233991 -lt 231928233990 2021-07-26T06:57:19.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:19.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509459 2021-07-26T06:57:19.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:19.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:57:19.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509459 2021-07-26T06:57:19.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:19.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509459 2021-07-26T06:57:19.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2021-07-26T06:57:19.915 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509459 2021-07-26T06:57:19.915 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:57:20.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509460 -lt 64424509459 2021-07-26T06:57:20.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:57:20.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:20.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:20.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:57:20.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:20.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:20.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:20.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:20.592 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:20.593 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:20.593 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:20.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:57:20.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:20.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:20.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:21.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:57:21.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:57:21.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:57:21.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3661: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2021-07-26T06:57:21.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:57:21.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:57:21.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=2 2021-07-26T06:57:21.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:57:21.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2021-07-26T06:57:21.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:57:21.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:57:21.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=2 2021-07-26T06:57:21.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:57:21.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2021-07-26T06:57:21.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:57:21.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:57:21.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:57:21.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:57:21.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:57:21.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:57:21.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2021-07-26T06:57:21.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:57:21.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:57:21.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=2 2021-07-26T06:57:21.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:57:21.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:57:21.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2021-07-26T06:57:22.047 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:57:22.047 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T06:57:22.047 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:57:22.047 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s1_head,1#3:00000000::::head#, (61) No data available 2021-07-26T06:57:22.048 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:57:22.048 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:57:22.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2021-07-26T06:57:22.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:57:22.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:57:22.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:57:22.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:57:22.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:57:22.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:57:22.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:57:22.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:57:22.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:57:22.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:57:22.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:57:22.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:57:22.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:57:22.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:57:22.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:57:22.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:57:22.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:57:22.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:57:22.510 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:57:22.511 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:57:22.511 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:57:22.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:57:22.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:57:22.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:57:22.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:57:22.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:57:22.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:57:22.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:57:22.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:57:22.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:57:22.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:57:22.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:57:22.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:57:22.514 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.2 2021-07-26T06:57:22.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:57:22.514 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/2/whoami 2021-07-26T06:57:22.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:57:22.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:57:22.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:57:22.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:57:22.525 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:22.525+0000 7efdb1f590c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:22.527 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:22.527+0000 7efdb1f590c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:22.528 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:22.528+0000 7efdb1f590c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:22.530 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:22.530+0000 7efdb1f590c0 -1 Falling back to public interface 2021-07-26T06:57:22.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:57:22.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:57:22.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:57:22.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:57:22.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:57:22.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:57:22.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:57:22.787 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:57:22.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:57:22.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:57:22.890 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:22.889+0000 7efdb1f590c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:57:22.897 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:22.897+0000 7efdb1f590c0 -1 journal do_read_entry(1441792): bad header magic 2021-07-26T06:57:22.898 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:22.897+0000 7efdb1f590c0 -1 journal do_read_entry(1441792): bad header magic 2021-07-26T06:57:22.905 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:22.904+0000 7efdb1f590c0 -1 osd.2 63 log_to_monitors {default=true} 2021-07-26T06:57:23.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:57:24.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:57:24.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:57:24.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:57:24.054 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:57:24.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:57:24.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:57:24.507 INFO:tasks.workunit.client.0.smithi071.stderr:osd.2 up in weight 1 up_from 67 up_thru 67 down_at 64 last_clean_interval [15,63) [v2:127.0.0.1:6818/307792,v1:127.0.0.1:6819/307792] [v2:127.0.0.1:6820/307792,v1:127.0.0.1:6821/307792] exists,up 8d19ee35-ef93-4bea-8512-476d49af717d 2021-07-26T06:57:24.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:57:24.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:57:24.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:57:24.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:57:24.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:57:24.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:57:24.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:57:24.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:57:24.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:57:24.510 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:57:24.510 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:57:24.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:57:24.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:57:24.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:57:24.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:57:24.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:57:24.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:57:24.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:57:24.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:57:24.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:57:24.872 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:57:24.873 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:57:24.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:57:24.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:24.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:57:24.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=266287972356 2021-07-26T06:57:24.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 266287972356 2021-07-26T06:57:24.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-266287972356' 2021-07-26T06:57:24.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:24.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:57:25.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=231928233993 2021-07-26T06:57:25.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 231928233993 2021-07-26T06:57:25.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-266287972356 1-231928233993' 2021-07-26T06:57:25.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:25.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:57:25.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=287762808834 2021-07-26T06:57:25.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 287762808834 2021-07-26T06:57:25.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-266287972356 1-231928233993 2-287762808834' 2021-07-26T06:57:25.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:25.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-266287972356 2021-07-26T06:57:25.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:25.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:57:25.257 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-266287972356 2021-07-26T06:57:25.257 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:25.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=266287972356 2021-07-26T06:57:25.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 266287972356' 2021-07-26T06:57:25.258 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 266287972356 2021-07-26T06:57:25.259 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:25.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 266287972355 -lt 266287972356 2021-07-26T06:57:25.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:57:26.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:57:26.524 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:26.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 266287972355 -lt 266287972356 2021-07-26T06:57:26.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:57:27.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:57:27.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:28.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 266287972357 -lt 266287972356 2021-07-26T06:57:28.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:28.077 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-231928233993 2021-07-26T06:57:28.077 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:28.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:57:28.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-231928233993 2021-07-26T06:57:28.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:28.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=231928233993 2021-07-26T06:57:28.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 231928233993' 2021-07-26T06:57:28.081 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 231928233993 2021-07-26T06:57:28.081 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:57:28.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 231928233993 -lt 231928233993 2021-07-26T06:57:28.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:28.385 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-287762808834 2021-07-26T06:57:28.385 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:28.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:57:28.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-287762808834 2021-07-26T06:57:28.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:28.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=287762808834 2021-07-26T06:57:28.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 287762808834' 2021-07-26T06:57:28.388 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 287762808834 2021-07-26T06:57:28.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:57:28.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 287762808834 -lt 287762808834 2021-07-26T06:57:28.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:57:28.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:28.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:28.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:57:28.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:28.982 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:28.982 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:28.982 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:28.982 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:28.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:28.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:29.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:57:29.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:29.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:29.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:29.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:57:29.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:57:29.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:57:29.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3662: corrupt_scrub_erasure: echo -n val3-EOBJ4 2021-07-26T06:57:29.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2021-07-26T06:57:29.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:57:29.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:57:29.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=2 2021-07-26T06:57:29.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:57:29.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2021-07-26T06:57:29.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:57:29.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:57:29.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=2 2021-07-26T06:57:29.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:57:29.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2021-07-26T06:57:29.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:57:29.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:57:29.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:57:29.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:57:29.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:57:29.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:57:29.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2021-07-26T06:57:29.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:57:29.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:57:29.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=2 2021-07-26T06:57:29.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:57:29.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:57:29.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2021-07-26T06:57:30.611 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:57:30.612 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T06:57:30.612 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:57:30.612 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s1_head,1#3:00000000::::head#, (61) No data available 2021-07-26T06:57:30.612 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:57:30.613 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:57:31.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2021-07-26T06:57:31.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:57:31.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:57:31.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:57:31.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:57:31.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:57:31.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:57:31.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:57:31.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:57:31.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:57:31.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:57:31.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:57:31.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:57:31.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:57:31.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:57:31.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:57:31.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:57:31.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:57:31.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:57:31.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:57:31.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:57:31.148 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:57:31.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:57:31.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:57:31.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:57:31.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:57:31.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:57:31.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:57:31.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:57:31.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:57:31.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:57:31.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:57:31.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:57:31.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:57:31.151 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.2 2021-07-26T06:57:31.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:57:31.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/2/whoami 2021-07-26T06:57:31.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:57:31.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:57:31.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:57:31.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:57:31.167 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:31.166+0000 7f5b50d3f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:31.169 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:31.169+0000 7f5b50d3f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:31.170 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:31.170+0000 7f5b50d3f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:31.171 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:31.171+0000 7f5b50d3f0c0 -1 Falling back to public interface 2021-07-26T06:57:31.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:57:31.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:57:31.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:57:31.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:57:31.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:57:31.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:57:31.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:57:31.561 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:57:31.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:57:31.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:57:31.764 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:31.764+0000 7f5b50d3f0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:57:31.772 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:31.772+0000 7f5b50d3f0c0 -1 journal do_read_entry(1634304): bad header magic 2021-07-26T06:57:31.772 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:31.772+0000 7f5b50d3f0c0 -1 journal do_read_entry(1634304): bad header magic 2021-07-26T06:57:31.783 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:31.782+0000 7f5b50d3f0c0 -1 osd.2 69 log_to_monitors {default=true} 2021-07-26T06:57:31.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:57:32.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:57:32.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:57:32.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:57:32.845 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:57:32.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:57:32.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:57:33.151 INFO:tasks.workunit.client.0.smithi071.stderr:osd.2 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [67,69) [v2:127.0.0.1:6818/308420,v1:127.0.0.1:6819/308420] [v2:127.0.0.1:6820/308420,v1:127.0.0.1:6821/308420] exists,up 8d19ee35-ef93-4bea-8512-476d49af717d 2021-07-26T06:57:33.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:57:33.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:57:33.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:57:33.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:57:33.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:57:33.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:57:33.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:57:33.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:57:33.154 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:57:33.154 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:57:33.154 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:57:33.154 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:57:33.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:57:33.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:57:33.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:57:33.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:57:33.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:57:33.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:57:33.251 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:57:33.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:57:33.513 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:57:33.513 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:57:33.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:57:33.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:33.514 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:57:33.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=266287972359 2021-07-26T06:57:33.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 266287972359 2021-07-26T06:57:33.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-266287972359' 2021-07-26T06:57:33.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:33.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:57:33.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=231928233996 2021-07-26T06:57:33.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 231928233996 2021-07-26T06:57:33.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-266287972359 1-231928233996' 2021-07-26T06:57:33.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:33.747 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:57:33.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=313532612610 2021-07-26T06:57:33.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 313532612610 2021-07-26T06:57:33.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-266287972359 1-231928233996 2-313532612610' 2021-07-26T06:57:33.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:33.864 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-266287972359 2021-07-26T06:57:33.864 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:33.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:57:33.865 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-266287972359 2021-07-26T06:57:33.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:33.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=266287972359 2021-07-26T06:57:33.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 266287972359' 2021-07-26T06:57:33.866 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 266287972359 2021-07-26T06:57:33.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:34.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 266287972358 -lt 266287972359 2021-07-26T06:57:34.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:57:35.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:57:35.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:35.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 266287972359 -lt 266287972359 2021-07-26T06:57:35.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:35.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-231928233996 2021-07-26T06:57:35.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:35.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:57:35.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-231928233996 2021-07-26T06:57:35.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:35.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=231928233996 2021-07-26T06:57:35.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 231928233996' 2021-07-26T06:57:35.416 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 231928233996 2021-07-26T06:57:35.416 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:57:35.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 231928233996 -lt 231928233996 2021-07-26T06:57:35.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:35.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-313532612610 2021-07-26T06:57:35.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:35.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:57:35.724 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-313532612610 2021-07-26T06:57:35.724 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:35.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=313532612610 2021-07-26T06:57:35.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2021-07-26T06:57:35.725 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 313532612610 2021-07-26T06:57:35.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:57:36.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 313532612610 -lt 313532612610 2021-07-26T06:57:36.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:57:36.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:36.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:36.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:57:36.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:36.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:36.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:36.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:36.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:36.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:36.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:36.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:57:36.699 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:36.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:36.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:37.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:57:37.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:57:37.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:57:37.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2021-07-26T06:57:37.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T06:57:37.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ5 2021-07-26T06:57:37.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2021-07-26T06:57:37.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:57:37.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T06:57:37.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ5 2021-07-26T06:57:37.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:57:37.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:57:37.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:57:37.552 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:57:37.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:57:37.954 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:57:37.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:57:37.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:57:37.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:57:38.128 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 5 % 2 2021-07-26T06:57:38.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=1 2021-07-26T06:57:38.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T06:57:38.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2021-07-26T06:57:38.132 INFO:tasks.workunit.client.0.smithi071.stderr:2+0 records in 2021-07-26T06:57:38.132 INFO:tasks.workunit.client.0.smithi071.stderr:2+0 records out 2021-07-26T06:57:38.133 INFO:tasks.workunit.client.0.smithi071.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 9.0308e-05 s, 45.4 MB/s 2021-07-26T06:57:38.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:57:38.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:57:38.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:57:38.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:57:38.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:57:38.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:57:38.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:57:38.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:57:38.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:57:38.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:57:38.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:57:38.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:57:38.136 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:57:38.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:57:38.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:57:38.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:57:38.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:57:38.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:57:38.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:57:38.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:57:38.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:57:38.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:57:38.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:57:38.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:57:39.120 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:57:39.120 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T06:57:39.121 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:57:39.121 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s0_head,0#3:00000000::::head#, (61) No data available 2021-07-26T06:57:39.121 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:57:39.121 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:57:39.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:57:39.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:57:39.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:57:39.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:57:39.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:57:39.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:57:39.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:57:39.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:57:39.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:57:39.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:57:39.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:57:39.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:57:39.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:57:39.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:57:39.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:57:39.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:57:39.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:57:39.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:57:39.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:57:39.559 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:57:39.559 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:57:39.560 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:57:39.560 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:57:39.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:57:39.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:57:39.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:57:39.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:57:39.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:57:39.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:57:39.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:57:39.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:57:39.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:57:39.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:57:39.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:57:39.562 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:57:39.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:57:39.563 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:57:39.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:57:39.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:57:39.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:57:39.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:57:39.578 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:39.578+0000 7fb25c3090c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:39.580 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:39.580+0000 7fb25c3090c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:39.581 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:39.581+0000 7fb25c3090c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:39.582 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:39.583+0000 7fb25c3090c0 -1 Falling back to public interface 2021-07-26T06:57:39.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:57:39.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:57:39.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:57:39.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:57:39.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:57:39.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:57:39.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:57:39.971 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:57:39.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:57:39.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:57:40.011 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:40.011+0000 7fb25c3090c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:57:40.022 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:40.022+0000 7fb25c3090c0 -1 journal do_read_entry(1937408): bad header magic 2021-07-26T06:57:40.023 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:40.022+0000 7fb25c3090c0 -1 journal do_read_entry(1937408): bad header magic 2021-07-26T06:57:40.030 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:40.030+0000 7fb25c3090c0 -1 osd.1 76 log_to_monitors {default=true} 2021-07-26T06:57:40.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:57:41.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:57:41.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:57:41.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:57:41.273 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:57:41.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:57:41.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:57:41.556 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 80 up_thru 80 down_at 77 last_clean_interval [54,76) [v2:127.0.0.1:6810/309093,v1:127.0.0.1:6811/309093] [v2:127.0.0.1:6812/309093,v1:127.0.0.1:6813/309093] exists,up bcf1ac15-08fb-47c5-9d33-a053d9f78c45 2021-07-26T06:57:41.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:57:41.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:57:41.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:57:41.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:57:41.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:57:41.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:57:41.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:57:41.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:57:41.558 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:57:41.558 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:57:41.559 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:57:41.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:57:41.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:57:41.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:57:41.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:57:41.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:57:41.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:57:41.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:57:41.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:57:41.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:57:41.922 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:57:41.922 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:57:41.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:57:41.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:41.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:57:42.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=266287972362 2021-07-26T06:57:42.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 266287972362 2021-07-26T06:57:42.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-266287972362' 2021-07-26T06:57:42.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:42.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:57:42.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=343597383682 2021-07-26T06:57:42.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 343597383682 2021-07-26T06:57:42.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-266287972362 1-343597383682' 2021-07-26T06:57:42.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:42.198 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:57:42.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=313532612613 2021-07-26T06:57:42.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 313532612613 2021-07-26T06:57:42.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-266287972362 1-343597383682 2-313532612613' 2021-07-26T06:57:42.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:42.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-266287972362 2021-07-26T06:57:42.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:42.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:57:42.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-266287972362 2021-07-26T06:57:42.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:42.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=266287972362 2021-07-26T06:57:42.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 266287972362' 2021-07-26T06:57:42.319 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 266287972362 2021-07-26T06:57:42.319 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:42.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 266287972360 -lt 266287972362 2021-07-26T06:57:42.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:57:43.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:57:43.583 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:43.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 266287972362 -lt 266287972362 2021-07-26T06:57:43.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:43.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-343597383682 2021-07-26T06:57:43.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:43.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:57:43.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-343597383682 2021-07-26T06:57:43.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:43.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=343597383682 2021-07-26T06:57:43.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2021-07-26T06:57:43.864 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 343597383682 2021-07-26T06:57:43.865 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:57:44.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 343597383682 -lt 343597383682 2021-07-26T06:57:44.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:44.131 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-313532612613 2021-07-26T06:57:44.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:44.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:57:44.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-313532612613 2021-07-26T06:57:44.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:44.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=313532612613 2021-07-26T06:57:44.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 313532612613' 2021-07-26T06:57:44.135 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 313532612613 2021-07-26T06:57:44.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:57:44.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 313532612613 -lt 313532612613 2021-07-26T06:57:44.567 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:57:44.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:44.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:44.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:57:44.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:44.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:44.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:44.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:44.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:44.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:44.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:45.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:57:45.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:45.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:45.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:45.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:57:45.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:57:45.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:57:45.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T06:57:45.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ6 2021-07-26T06:57:45.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2021-07-26T06:57:45.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:57:45.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T06:57:45.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ6 2021-07-26T06:57:45.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:57:45.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:57:45.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:57:45.837 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:57:45.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:57:46.297 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:57:46.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:57:46.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:57:46.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:57:55.169 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 6 % 2 2021-07-26T06:57:55.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=0 2021-07-26T06:57:55.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T06:57:55.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3674: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2021-07-26T06:57:55.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:57:55.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:57:55.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:57:55.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:57:55.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2021-07-26T06:57:55.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:57:55.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:57:55.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:57:55.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:57:55.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:57:55.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:57:55.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:57:55.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:57:55.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:57:55.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:57:55.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:57:55.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2021-07-26T06:57:55.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:57:55.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:57:55.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:57:55.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:57:55.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:57:55.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2021-07-26T06:57:56.085 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:57:56.085 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T06:57:56.086 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s2_head,2#3:00000000::::head#, (61) No data available 2021-07-26T06:57:56.086 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:57:56.086 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:57:56.086 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:57:56.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T06:57:56.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:57:56.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:57:56.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:57:56.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:57:56.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:57:56.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:57:56.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:57:56.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:57:56.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:57:56.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:57:56.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:57:56.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:57:56.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:57:56.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:57:56.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:57:56.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:57:56.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:57:56.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:57:56.555 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:57:56.555 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:57:56.555 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:57:56.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:57:56.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:57:56.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:57:56.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:57:56.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:57:56.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:57:56.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:57:56.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:57:56.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:57:56.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:57:56.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:57:56.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:57:56.558 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:57:56.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:57:56.558 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:57:56.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:57:56.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:57:56.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:57:56.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:57:56.574 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:56.572+0000 7f1a36abd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:56.575 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:56.574+0000 7f1a36abd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:56.577 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:56.576+0000 7f1a36abd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:57:56.578 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:56.577+0000 7f1a36abd0c0 -1 Falling back to public interface 2021-07-26T06:57:56.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:57:56.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:57:56.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:57:56.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:57:56.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:57:56.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:57:56.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:57:56.833 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:57:56.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:57:56.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:57:56.920 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:56.918+0000 7f1a36abd0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:57:56.936 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:56.935+0000 7f1a36abd0c0 -1 journal do_read_entry(2138112): bad header magic 2021-07-26T06:57:56.937 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:56.935+0000 7f1a36abd0c0 -1 journal do_read_entry(2138112): bad header magic 2021-07-26T06:57:56.944 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:57:56.943+0000 7f1a36abd0c0 -1 osd.0 84 log_to_monitors {default=true} 2021-07-26T06:57:57.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:57:58.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:57:58.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:57:58.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:57:58.202 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:57:58.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:57:58.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:57:58.512 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [62,84) [v2:127.0.0.1:6800/309764,v1:127.0.0.1:6801/309764] [v2:127.0.0.1:6802/309764,v1:127.0.0.1:6803/309764] exists,up 02a6c4ed-978b-40d7-8fc6-dfa7d25e4130 2021-07-26T06:57:58.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:57:58.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:57:58.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:57:58.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:57:58.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:57:58.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:57:58.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:57:58.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:57:58.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:57:58.515 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:57:58.515 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:57:58.516 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:57:58.516 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:57:58.516 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:57:58.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:57:58.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:57:58.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:57:58.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:57:58.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:57:58.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:57:58.874 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:57:58.874 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:57:58.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:57:58.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:58.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:57:59.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=377957122050 2021-07-26T06:57:59.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 377957122050 2021-07-26T06:57:59.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-377957122050' 2021-07-26T06:57:59.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:59.009 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:57:59.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=343597383686 2021-07-26T06:57:59.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 343597383686 2021-07-26T06:57:59.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-377957122050 1-343597383686' 2021-07-26T06:57:59.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:59.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:57:59.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=313532612617 2021-07-26T06:57:59.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 313532612617 2021-07-26T06:57:59.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-377957122050 1-343597383686 2-313532612617' 2021-07-26T06:57:59.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:59.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-377957122050 2021-07-26T06:57:59.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:59.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:57:59.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-377957122050 2021-07-26T06:57:59.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:59.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=377957122050 2021-07-26T06:57:59.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 377957122050' 2021-07-26T06:57:59.271 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 377957122050 2021-07-26T06:57:59.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:59.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 377957122050 2021-07-26T06:57:59.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:58:00.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:58:00.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:58:00.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 377957122050 2021-07-26T06:58:00.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:58:01.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:58:01.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:58:02.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 377957122050 -lt 377957122050 2021-07-26T06:58:02.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:58:02.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-343597383686 2021-07-26T06:58:02.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:58:02.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:58:02.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-343597383686 2021-07-26T06:58:02.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:58:02.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=343597383686 2021-07-26T06:58:02.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 343597383686' 2021-07-26T06:58:02.097 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 343597383686 2021-07-26T06:58:02.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:58:02.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 343597383687 -lt 343597383686 2021-07-26T06:58:02.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:58:02.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-313532612617 2021-07-26T06:58:02.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:58:02.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:58:02.405 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-313532612617 2021-07-26T06:58:02.405 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:58:02.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=313532612617 2021-07-26T06:58:02.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 313532612617' 2021-07-26T06:58:02.407 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 313532612617 2021-07-26T06:58:02.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:58:02.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 313532612617 -lt 313532612617 2021-07-26T06:58:02.673 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:58:02.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:02.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:03.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:58:03.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:03.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:03.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:03.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:03.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:03.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:03.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:03.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:58:03.306 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:03.306 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:03.306 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:03.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:58:03.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:58:03.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:58:03.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3675: corrupt_scrub_erasure: echo -n bad-val 2021-07-26T06:58:03.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2021-07-26T06:58:03.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:58:03.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:58:03.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:58:03.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:58:03.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2021-07-26T06:58:03.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:58:03.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:58:03.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:58:03.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:58:03.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:58:03.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:58:03.637 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:58:03.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:58:03.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:58:03.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:58:03.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:58:03.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2021-07-26T06:58:03.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:58:03.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:58:03.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:58:03.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:58:03.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:58:03.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2021-07-26T06:58:04.528 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:58:04.529 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T06:58:04.529 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:58:04.529 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s0_head,0#3:00000000::::head#, (61) No data available 2021-07-26T06:58:04.529 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:58:04.530 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:58:04.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:58:05.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:58:05.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:58:05.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:58:05.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:58:05.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:58:05.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:58:05.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:58:05.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:58:05.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:58:05.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:58:05.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:58:05.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:58:05.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:58:05.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:58:05.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:58:05.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:58:05.030 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:58:05.030 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:58:05.030 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:58:05.030 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:58:05.031 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:58:05.031 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:58:05.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:58:05.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:58:05.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:58:05.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:58:05.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:58:05.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:58:05.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:58:05.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:58:05.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:58:05.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:58:05.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:58:05.033 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:58:05.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:58:05.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:58:05.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:58:05.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:58:05.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:58:05.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:58:05.036 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:05.011+0000 7f1bddf350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:05.036 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:05.013+0000 7f1bddf350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:05.037 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:05.014+0000 7f1bddf350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:05.037 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:05.015+0000 7f1bddf350c0 -1 Falling back to public interface 2021-07-26T06:58:05.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:58:05.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:58:05.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:58:05.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:58:05.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:58:05.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:58:05.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:58:05.268 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:58:05.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:58:05.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:58:05.526 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:05.525+0000 7f1bddf350c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:58:05.527 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:05.525+0000 7f1bddf350c0 -1 journal do_read_entry(2449408): bad header magic 2021-07-26T06:58:05.527 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:05.525+0000 7f1bddf350c0 -1 journal do_read_entry(2449408): bad header magic 2021-07-26T06:58:05.534 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:05.533+0000 7f1bddf350c0 -1 osd.1 90 log_to_monitors {default=true} 2021-07-26T06:58:05.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:58:06.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:58:06.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:58:06.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:58:06.571 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:58:06.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:58:06.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:58:06.958 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 94 up_thru 94 down_at 91 last_clean_interval [80,90) [v2:127.0.0.1:6810/310392,v1:127.0.0.1:6811/310392] [v2:127.0.0.1:6812/310392,v1:127.0.0.1:6813/310392] exists,up bcf1ac15-08fb-47c5-9d33-a053d9f78c45 2021-07-26T06:58:06.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:58:06.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:58:06.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:58:06.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:58:06.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:58:06.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:58:06.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:58:06.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:58:06.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:58:06.961 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:58:06.961 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:58:06.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:58:06.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:58:06.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:58:07.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:58:07.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:58:07.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:58:07.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:58:07.055 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:58:07.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:58:07.349 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:58:07.349 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:58:07.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:58:07.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:58:07.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:58:07.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=377957122053 2021-07-26T06:58:07.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 377957122053 2021-07-26T06:58:07.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-377957122053' 2021-07-26T06:58:07.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:58:07.483 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:58:07.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=403726925826 2021-07-26T06:58:07.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 403726925826 2021-07-26T06:58:07.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-377957122053 1-403726925826' 2021-07-26T06:58:07.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:58:07.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:58:07.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=313532612620 2021-07-26T06:58:07.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 313532612620 2021-07-26T06:58:07.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-377957122053 1-403726925826 2-313532612620' 2021-07-26T06:58:07.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:58:07.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-377957122053 2021-07-26T06:58:07.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:58:07.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:58:07.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-377957122053 2021-07-26T06:58:07.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:58:07.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=377957122053 2021-07-26T06:58:07.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 377957122053' 2021-07-26T06:58:07.721 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 377957122053 2021-07-26T06:58:07.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:58:07.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 377957122051 -lt 377957122053 2021-07-26T06:58:07.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:58:08.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:58:08.985 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:58:09.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 377957122053 -lt 377957122053 2021-07-26T06:58:09.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:58:09.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-403726925826 2021-07-26T06:58:09.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:58:09.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:58:09.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-403726925826 2021-07-26T06:58:09.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:58:09.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=403726925826 2021-07-26T06:58:09.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 403726925826' 2021-07-26T06:58:09.263 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 403726925826 2021-07-26T06:58:09.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:58:09.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 403726925826 -lt 403726925826 2021-07-26T06:58:09.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:58:09.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-313532612620 2021-07-26T06:58:09.534 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:58:09.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:58:09.537 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-313532612620 2021-07-26T06:58:09.537 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:58:09.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=313532612620 2021-07-26T06:58:09.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 313532612620' 2021-07-26T06:58:09.538 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 313532612620 2021-07-26T06:58:09.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:58:09.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 313532612620 -lt 313532612620 2021-07-26T06:58:09.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:58:09.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:09.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:10.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:58:10.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:10.172 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:10.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:10.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:10.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:10.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:10.174 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:10.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:58:10.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:10.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:10.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:10.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:58:10.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:58:10.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:58:10.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T06:58:10.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ7 2021-07-26T06:58:10.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2021-07-26T06:58:10.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:58:10.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T06:58:10.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ7 2021-07-26T06:58:10.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:58:10.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:58:10.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:58:11.273 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:58:11.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:58:11.791 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:58:11.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:58:11.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:58:11.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:58:20.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 7 % 2 2021-07-26T06:58:20.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=1 2021-07-26T06:58:20.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T06:58:20.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2021-07-26T06:58:20.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3681: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2021-07-26T06:58:20.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2021-07-26T06:58:20.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3685: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2021-07-26T06:58:20.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:58:20.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:58:20.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:58:20.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:58:20.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2021-07-26T06:58:20.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:58:20.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:58:20.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:58:20.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:58:20.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:58:20.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:58:20.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:58:20.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:58:20.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:58:20.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:58:20.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:58:20.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2021-07-26T06:58:20.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:58:20.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:58:20.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:58:20.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:58:20.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:58:20.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2021-07-26T06:58:22.023 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:58:22.023 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T06:58:22.023 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s2_head,2#3:00000000::::head#, (61) No data available 2021-07-26T06:58:22.024 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:58:22.024 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:58:22.024 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:58:22.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T06:58:22.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:58:22.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:58:22.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:58:22.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:58:22.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:58:22.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:58:22.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:58:22.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:58:22.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:58:22.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:58:22.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:58:22.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:58:22.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:58:22.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:58:22.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:58:22.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:58:22.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:58:22.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:58:22.341 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:58:22.342 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:58:22.342 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:58:22.342 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:58:22.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:58:22.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:58:22.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:58:22.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:58:22.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:58:22.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:58:22.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:58:22.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:58:22.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:58:22.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:58:22.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:58:22.345 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:58:22.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:58:22.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:58:22.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:58:22.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:58:22.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:58:22.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:58:22.360 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:22.359+0000 7fa5b2d2c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:22.362 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:22.362+0000 7fa5b2d2c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:22.363 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:22.363+0000 7fa5b2d2c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:22.364 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:22.364+0000 7fa5b2d2c0c0 -1 Falling back to public interface 2021-07-26T06:58:22.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:58:22.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:58:22.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:58:22.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:58:22.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:58:22.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:58:22.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:58:22.620 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:58:22.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:58:22.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:58:22.741 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:22.741+0000 7fa5b2d2c0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:58:22.749 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:22.749+0000 7fa5b2d2c0c0 -1 journal do_read_entry(2605056): bad header magic 2021-07-26T06:58:22.750 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:22.749+0000 7fa5b2d2c0c0 -1 journal do_read_entry(2605056): bad header magic 2021-07-26T06:58:22.757 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:22.756+0000 7fa5b2d2c0c0 -1 osd.0 98 log_to_monitors {default=true} 2021-07-26T06:58:22.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:58:23.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:58:23.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:58:23.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:58:23.941 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:58:23.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:58:23.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:58:24.215 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 102 up_thru 102 down_at 99 last_clean_interval [88,98) [v2:127.0.0.1:6800/311091,v1:127.0.0.1:6801/311091] [v2:127.0.0.1:6802/311091,v1:127.0.0.1:6803/311091] exists,up 02a6c4ed-978b-40d7-8fc6-dfa7d25e4130 2021-07-26T06:58:24.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:58:24.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:58:24.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:58:24.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:58:24.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:58:24.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:58:24.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:58:24.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:58:24.218 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:58:24.218 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:58:24.218 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:58:24.218 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:58:24.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:58:24.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:58:24.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:58:24.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:58:24.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:58:24.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:58:24.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:58:24.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:58:24.578 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:58:24.578 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:58:24.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:58:24.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:58:24.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:58:24.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=438086664194 2021-07-26T06:58:24.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 438086664194 2021-07-26T06:58:24.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-438086664194' 2021-07-26T06:58:24.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:58:24.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:58:24.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=403726925830 2021-07-26T06:58:24.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 403726925830 2021-07-26T06:58:24.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-438086664194 1-403726925830' 2021-07-26T06:58:24.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:58:24.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:58:24.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=313532612624 2021-07-26T06:58:24.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 313532612624 2021-07-26T06:58:24.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-438086664194 1-403726925830 2-313532612624' 2021-07-26T06:58:24.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:58:24.930 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-438086664194 2021-07-26T06:58:24.930 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:58:24.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:58:24.932 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-438086664194 2021-07-26T06:58:24.932 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:58:24.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=438086664194 2021-07-26T06:58:24.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 438086664194' 2021-07-26T06:58:24.933 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 438086664194 2021-07-26T06:58:24.933 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:58:25.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 438086664194 -lt 438086664194 2021-07-26T06:58:25.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:58:25.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-403726925830 2021-07-26T06:58:25.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:58:25.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:58:25.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-403726925830 2021-07-26T06:58:25.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:58:25.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=403726925830 2021-07-26T06:58:25.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 403726925830' 2021-07-26T06:58:25.207 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 403726925830 2021-07-26T06:58:25.207 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:58:25.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 403726925830 -lt 403726925830 2021-07-26T06:58:25.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:58:25.537 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-313532612624 2021-07-26T06:58:25.537 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:58:25.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:58:25.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-313532612624 2021-07-26T06:58:25.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:58:25.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=313532612624 2021-07-26T06:58:25.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 313532612624' 2021-07-26T06:58:25.539 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 313532612624 2021-07-26T06:58:25.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:58:25.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 313532612623 -lt 313532612624 2021-07-26T06:58:25.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:58:26.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:58:26.782 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:58:27.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 313532612624 -lt 313532612624 2021-07-26T06:58:27.049 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:58:27.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:27.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:27.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:58:27.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:27.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:27.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:27.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:27.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:27.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:27.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:27.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:58:27.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:27.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:27.697 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:28.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:58:28.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:58:28.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:58:28.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2021-07-26T06:58:28.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:58:28.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:58:28.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:58:28.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:58:28.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2021-07-26T06:58:28.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:58:28.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:58:28.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:58:28.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:58:28.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:58:28.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:58:28.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:58:28.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:58:28.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:58:28.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:58:28.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:58:28.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2021-07-26T06:58:28.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:58:28.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:58:28.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:58:28.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:58:28.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:58:28.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2021-07-26T06:58:29.023 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:58:29.024 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T06:58:29.024 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s2_head,2#3:00000000::::head#, (61) No data available 2021-07-26T06:58:29.025 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:58:29.025 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:58:29.026 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:58:29.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T06:58:29.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:58:29.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:58:29.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:58:29.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:58:29.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:58:29.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:58:29.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:58:29.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:58:29.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:58:29.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:58:29.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:58:29.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:58:29.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:58:29.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:58:29.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:58:29.471 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:58:29.471 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:58:29.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:58:29.472 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:58:29.472 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:58:29.472 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:58:29.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:58:29.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:58:29.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:58:29.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:58:29.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:58:29.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:58:29.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:58:29.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:58:29.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:58:29.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:58:29.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:58:29.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:58:29.476 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:58:29.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:58:29.476 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:58:29.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:58:29.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:58:29.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:58:29.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:58:29.493 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:29.491+0000 7f451f4c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:29.493 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:29.493+0000 7f451f4c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:29.494 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:29.494+0000 7f451f4c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:29.495 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:29.495+0000 7f451f4c50c0 -1 Falling back to public interface 2021-07-26T06:58:29.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:58:29.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:58:29.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:58:29.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:58:29.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:58:29.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:58:29.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:58:29.812 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:58:29.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:58:29.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:58:29.937 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:29.935+0000 7f451f4c50c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:58:29.944 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:29.944+0000 7f451f4c50c0 -1 journal do_read_entry(2813952): bad header magic 2021-07-26T06:58:29.945 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:29.944+0000 7f451f4c50c0 -1 journal do_read_entry(2813952): bad header magic 2021-07-26T06:58:29.952 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:29.952+0000 7f451f4c50c0 -1 osd.0 104 log_to_monitors {default=true} 2021-07-26T06:58:30.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:58:31.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:58:31.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:58:31.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:58:31.134 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:58:31.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:58:31.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:58:31.421 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 108 up_thru 108 down_at 105 last_clean_interval [102,104) [v2:127.0.0.1:6800/311695,v1:127.0.0.1:6801/311695] [v2:127.0.0.1:6802/311695,v1:127.0.0.1:6803/311695] exists,up 02a6c4ed-978b-40d7-8fc6-dfa7d25e4130 2021-07-26T06:58:31.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:58:31.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:58:31.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:58:31.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:58:31.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:58:31.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:58:31.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:58:31.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:58:31.423 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:58:31.424 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:58:31.424 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:58:31.424 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:58:31.424 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:58:31.424 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:58:31.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:58:31.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:58:31.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:58:31.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:58:31.519 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:58:31.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:58:31.783 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:58:31.783 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:58:31.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:58:31.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:58:31.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:58:31.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=463856467970 2021-07-26T06:58:31.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 463856467970 2021-07-26T06:58:31.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-463856467970' 2021-07-26T06:58:31.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:58:31.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:58:32.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=403726925833 2021-07-26T06:58:32.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 403726925833 2021-07-26T06:58:32.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-463856467970 1-403726925833' 2021-07-26T06:58:32.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:58:32.059 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:58:32.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=313532612627 2021-07-26T06:58:32.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 313532612627 2021-07-26T06:58:32.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-463856467970 1-403726925833 2-313532612627' 2021-07-26T06:58:32.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:58:32.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-463856467970 2021-07-26T06:58:32.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:58:32.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:58:32.178 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-463856467970 2021-07-26T06:58:32.178 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:58:32.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=463856467970 2021-07-26T06:58:32.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 463856467970' 2021-07-26T06:58:32.179 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 463856467970 2021-07-26T06:58:32.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:58:32.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 463856467970 2021-07-26T06:58:32.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:58:33.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:58:33.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:58:33.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 463856467970 -lt 463856467970 2021-07-26T06:58:33.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:58:33.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-403726925833 2021-07-26T06:58:33.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:58:33.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:58:33.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-403726925833 2021-07-26T06:58:33.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:58:33.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=403726925833 2021-07-26T06:58:33.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 403726925833' 2021-07-26T06:58:33.723 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 403726925833 2021-07-26T06:58:33.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:58:34.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 403726925833 -lt 403726925833 2021-07-26T06:58:34.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:58:34.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-313532612627 2021-07-26T06:58:34.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:58:34.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:58:34.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-313532612627 2021-07-26T06:58:34.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:58:34.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=313532612627 2021-07-26T06:58:34.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 313532612627' 2021-07-26T06:58:34.028 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 313532612627 2021-07-26T06:58:34.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:58:34.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 313532612627 -lt 313532612627 2021-07-26T06:58:34.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:58:34.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:34.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:34.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:58:34.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:34.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:34.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:34.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:34.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:34.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:34.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:34.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:58:34.957 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:34.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:34.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:35.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:58:35.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:58:35.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:58:35.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3687: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2021-07-26T06:58:35.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3693: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2021-07-26T06:58:35.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T06:58:35.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=EOBJ0 2021-07-26T06:58:35.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool EOBJ0 2021-07-26T06:58:35.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:58:35.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3693: corrupt_scrub_erasure: local pg=3.0 2021-07-26T06:58:35.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3695: corrupt_scrub_erasure: pg_scrub 3.0 2021-07-26T06:58:35.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: pg_scrub: local pgid=3.0 2021-07-26T06:58:35.628 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: pg_scrub: get_last_scrub_stamp 3.0 2021-07-26T06:58:35.628 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T06:58:35.628 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:58:35.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:58:35.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T06:58:35.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: pg_scrub: local last_scrub=2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:35.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: pg_scrub: ceph pg scrub 3.0 2021-07-26T06:58:36.184 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 3.0s0 on osd.1 to scrub 2021-07-26T06:58:36.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: pg_scrub: wait_for_scrub 3.0 2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:36.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=3.0 2021-07-26T06:58:36.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:36.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T06:58:36.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:58:36.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:58:36.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T06:58:36.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T06:58:36.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:58:36.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:58:36.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T06:58:36.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:56:19.343467+0000 '>' 2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:36.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:58:37.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:58:37.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:58:37.485 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T06:58:37.485 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T06:58:37.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:58:37.488 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:58:37.488 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T06:58:37.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:56:19.343467+0000 '>' 2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:37.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:58:38.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:58:38.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:58:38.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T06:58:38.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T06:58:38.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:58:38.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:58:38.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T06:58:39.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:56:19.343467+0000 '>' 2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:39.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:58:40.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:58:40.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:58:40.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T06:58:40.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T06:58:40.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:58:40.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:58:40.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T06:58:40.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:56:19.343467+0000 '>' 2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:40.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:58:41.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:58:41.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:58:41.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T06:58:41.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T06:58:41.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T06:58:41.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:58:41.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T06:58:41.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:58:37.647794+0000 '>' 2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:41.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:58:41.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3697: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2021-07-26T06:58:41.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3699: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2021-07-26T06:58:41.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3699: corrupt_scrub_erasure: test 1 = 1 2021-07-26T06:58:41.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3701: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2021-07-26T06:58:41.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3701: corrupt_scrub_erasure: test 3.0 = 3.0 2021-07-26T06:58:41.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3703: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2021-07-26T06:58:41.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3705: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2021-07-26T06:58:41.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3705: corrupt_scrub_erasure: epoch=108 2021-07-26T06:58:41.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3707: corrupt_scrub_erasure: jq 'def walk(f): 2021-07-26T06:58:41.637 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T06:58:41.638 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T06:58:41.638 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T06:58:41.638 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T06:58:41.639 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T06:58:41.639 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T06:58:41.639 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T06:58:41.639 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T06:58:41.640 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T06:58:41.640 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T06:58:41.640 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T06:58:41.640 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2021-07-26T06:58:41.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3707: corrupt_scrub_erasure: jq .inconsistents 2021-07-26T06:58:41.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3707: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T06:58:41.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4327: corrupt_scrub_erasure: jq 'def walk(f): 2021-07-26T06:58:41.662 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T06:58:41.662 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T06:58:41.663 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T06:58:41.663 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T06:58:41.663 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T06:58:41.663 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T06:58:41.664 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T06:58:41.664 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T06:58:41.664 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T06:58:41.664 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T06:58:41.665 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T06:58:41.665 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2021-07-26T06:58:41.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4327: corrupt_scrub_erasure: jq .inconsistents 2021-07-26T06:58:41.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4327: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T06:58:41.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T06:58:41.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T06:58:41.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4329: corrupt_scrub_erasure: test no = yes 2021-07-26T06:58:41.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: test '' = yes 2021-07-26T06:58:41.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4339: corrupt_scrub_erasure: pg_deep_scrub 3.0 2021-07-26T06:58:41.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: pg_deep_scrub: local pgid=3.0 2021-07-26T06:58:41.689 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T06:58:41.689 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T06:58:41.689 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T06:58:41.689 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:58:41.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T06:58:41.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: pg_deep_scrub: local last_scrub=2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:41.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1800: pg_deep_scrub: ceph pg deep-scrub 3.0 2021-07-26T06:58:42.200 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2021-07-26T06:58:42.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: pg_deep_scrub: wait_for_scrub 3.0 2021-07-26T06:56:19.343467+0000 last_deep_scrub_stamp 2021-07-26T06:58:42.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=3.0 2021-07-26T06:58:42.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:42.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_deep_scrub_stamp 2021-07-26T06:58:42.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T06:58:42.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:58:42.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T06:58:42.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T06:58:42.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T06:58:42.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:58:42.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T06:58:42.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:56:19.343467+0000 '>' 2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:42.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:58:43.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:58:43.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:58:43.483 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T06:58:43.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T06:58:43.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T06:58:43.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:58:43.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T06:58:43.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:56:19.343467+0000 '>' 2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:43.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:58:44.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:58:44.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:58:44.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T06:58:44.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T06:58:44.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T06:58:44.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:58:44.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T06:58:45.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:56:19.343467+0000 '>' 2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:45.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:58:46.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:58:46.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:58:46.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T06:58:46.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T06:58:46.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T06:58:46.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:58:46.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T06:58:46.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:56:19.343467+0000 '>' 2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:46.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T06:58:47.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T06:58:47.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T06:58:47.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T06:58:47.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T06:58:47.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T06:58:47.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T06:58:47.308 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T06:58:47.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:58:42.630306+0000 '>' 2021-07-26T06:56:19.343467+0000 2021-07-26T06:58:47.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T06:58:47.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4341: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2021-07-26T06:58:47.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4343: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2021-07-26T06:58:47.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4343: corrupt_scrub_erasure: test 1 = 1 2021-07-26T06:58:47.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4345: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2021-07-26T06:58:47.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4345: corrupt_scrub_erasure: test 3.0 = 3.0 2021-07-26T06:58:47.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4347: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2021-07-26T06:58:47.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4349: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2021-07-26T06:58:47.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4349: corrupt_scrub_erasure: epoch=108 2021-07-26T06:58:47.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4351: corrupt_scrub_erasure: '[' false = true ']' 2021-07-26T06:58:47.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5008: corrupt_scrub_erasure: jq 'def walk(f): 2021-07-26T06:58:47.636 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T06:58:47.636 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T06:58:47.636 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T06:58:47.636 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T06:58:47.636 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T06:58:47.637 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T06:58:47.637 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T06:58:47.637 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T06:58:47.637 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T06:58:47.638 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T06:58:47.638 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T06:58:47.638 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2021-07-26T06:58:47.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5008: corrupt_scrub_erasure: jq .inconsistents 2021-07-26T06:58:47.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5008: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T06:58:47.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5741: corrupt_scrub_erasure: jq 'def walk(f): 2021-07-26T06:58:47.660 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T06:58:47.660 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T06:58:47.661 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T06:58:47.661 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T06:58:47.661 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T06:58:47.662 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T06:58:47.662 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T06:58:47.662 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T06:58:47.662 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T06:58:47.663 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T06:58:47.663 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T06:58:47.663 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2021-07-26T06:58:47.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5741: corrupt_scrub_erasure: jq .inconsistents 2021-07-26T06:58:47.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5741: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T06:58:47.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T06:58:47.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T06:58:47.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5743: corrupt_scrub_erasure: test no = yes 2021-07-26T06:58:47.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5754: corrupt_scrub_erasure: test '' = yes 2021-07-26T06:58:47.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2021-07-26T06:58:48.069 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' removed 2021-07-26T06:58:48.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T06:58:48.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:58:48.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:58:48.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:58:48.081 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:58:48.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:58:48.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:58:48.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:58:48.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:58:48.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:58:48.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:58:48.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:58:48.190 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:58:48.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:58:48.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:58:48.191 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:58:48.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:58:48.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:58:48.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:58:48.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:58:48.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:58:48.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:58:48.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:58:48.220 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:58:48.220 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:58:48.220 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:58:48.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:58:48.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:58:48.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:58:48.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T06:58:48.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T06:58:48.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T06:58:48.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T06:58:48.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T06:58:48.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:58:48.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T06:58:48.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:58:48.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:58:48.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:58:48.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:58:48.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:58:48.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:58:48.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:58:48.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:58:48.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:58:48.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:58:48.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:58:48.228 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:58:48.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:58:48.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:58:48.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:58:48.230 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:58:48.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:58:48.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:58:48.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T06:58:48.233 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:58:48.233 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:58:48.233 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:58:48.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T06:58:48.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:58:48.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:58:48.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T06:58:48.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:58:48.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:58:48.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:58:48.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T06:58:48.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:58:48.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:58:48.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:58:48.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T06:58:48.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2021-07-26T06:58:48.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2021-07-26T06:58:48.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5768: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2021-07-26T06:58:48.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2021-07-26T06:58:48.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: local allow_overwrites=true 2021-07-26T06:58:48.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3611: corrupt_scrub_erasure: local poolname=ecpool 2021-07-26T06:58:48.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: local total_objs=7 2021-07-26T06:58:48.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2021-07-26T06:58:48.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T06:58:48.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:58:48.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:58:48.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:58:48.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T06:58:48.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T06:58:48.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:58:48.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:58:48.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:58:48.609 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:58:48.609 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:58:48.610 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:58:48.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:58:48.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:58:48.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:58:48.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:58:48.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:58:48.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:58:48.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:58:48.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:58:48.862 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:58:48.862 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:58:48.863 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:58:48.863 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:58:48.863 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:58:48.864 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:58:48.864 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:58:48.865 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:58:48.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T06:58:48.959 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:58:48.959 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:58:48.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:58:48.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:58:48.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:58:48.960 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:58:48.960 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:58:48.961 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:58:48.961 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:58:48.961 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:58:48.961 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:58:48.962 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T06:58:48.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:58:48.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T06:58:49.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3615: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2021-07-26T06:58:49.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T06:58:49.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:58:49.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:58:49.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:58:49.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T06:58:49.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:58:49.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:58:49.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:58:49.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:58:49.890 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:58:49.891 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:58:49.891 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:58:49.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:58:49.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:58:49.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:58:49.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: seq 0 2 2021-07-26T06:58:49.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: for id in $(seq 0 2) 2021-07-26T06:58:49.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3617: corrupt_scrub_erasure: '[' true = true ']' 2021-07-26T06:58:49.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2021-07-26T06:58:49.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:58:49.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:58:49.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:58:49.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:58:49.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:58:49.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:58:49.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:58:49.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:58:49.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:58:49.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:58:49.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:58:49.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:58:49.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:58:49.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:58:49.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:58:49.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:58:49.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:58:49.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:58:49.923 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:58:49.923 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:58:49.923 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:58:49.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:58:49.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:58:49.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:58:49.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:58:49.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:58:49.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:58:49.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:58:49.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:58:49.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:58:49.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:58:49.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:58:49.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:58:49.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:58:49.926 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:58:49.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c3ed524e-cfd1-4ff6-b405-a9438f239b1a 2021-07-26T06:58:49.927 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 c3ed524e-cfd1-4ff6-b405-a9438f239b1a 2021-07-26T06:58:49.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c3ed524e-cfd1-4ff6-b405-a9438f239b1a' 2021-07-26T06:58:49.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:58:49.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQApXf5gd5KBNxAAsYnRBWnzErgFDDEFZ9zLow== 2021-07-26T06:58:49.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQApXf5gd5KBNxAAsYnRBWnzErgFDDEFZ9zLow=="}' 2021-07-26T06:58:49.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c3ed524e-cfd1-4ff6-b405-a9438f239b1a -i td/osd-scrub-repair/0/new.json 2021-07-26T06:58:50.227 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:58:50.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T06:58:50.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQApXf5gd5KBNxAAsYnRBWnzErgFDDEFZ9zLow== --osd-uuid c3ed524e-cfd1-4ff6-b405-a9438f239b1a 2021-07-26T06:58:50.262 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:50.261+0000 7f1607e690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:50.264 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:50.262+0000 7f1607e690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:50.264 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:50.263+0000 7f1607e690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:50.265 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:50.264+0000 7f1607e690c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:58:50.265 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:50.264+0000 7f1607e690c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:58:50.266 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:50.264+0000 7f1607e690c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T06:58:50.351 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:50.350+0000 7f1607e690c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T06:58:55.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T06:58:55.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:58:55.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:58:55.951 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T06:58:55.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:58:56.295 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T06:58:56.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:58:56.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:58:56.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:58:56.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:58:56.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:58:56.317 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:56.315+0000 7f3833e990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:56.318 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:56.317+0000 7f3833e990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:56.319 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:56.318+0000 7f3833e990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:58:56.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:58:56.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:58:56.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:58:56.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:58:56.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:58:56.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:58:56.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:58:56.568 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:58:56.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:58:56.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:58:56.846 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:56.845+0000 7f3833e990c0 -1 Falling back to public interface 2021-07-26T06:58:56.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:58:57.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:58:57.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:58:57.852 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:58:57.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:58:57.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:58:57.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:58:58.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:58:59.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:58:59.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:58:59.128 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:58:59.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:58:59.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:58:59.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:58:59.234 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:58:59.233+0000 7f3833e990c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:58:59.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:59:00.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:59:00.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:00.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:59:00.419 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:59:00.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:00.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:59:00.689 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/313878,v1:127.0.0.1:6803/313878] [v2:127.0.0.1:6804/313878,v1:127.0.0.1:6805/313878] exists,up c3ed524e-cfd1-4ff6-b405-a9438f239b1a 2021-07-26T06:59:00.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:59:00.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:59:00.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:59:00.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: for id in $(seq 0 2) 2021-07-26T06:59:00.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3617: corrupt_scrub_erasure: '[' true = true ']' 2021-07-26T06:59:00.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2021-07-26T06:59:00.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:59:00.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:59:00.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:59:00.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:59:00.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:59:00.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:59:00.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:59:00.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:59:00.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:59:00.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:59:00.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:59:00.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:59:00.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:59:00.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:59:00.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:59:00.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:59:00.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:59:00.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:59:00.700 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:59:00.700 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:59:00.701 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:59:00.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:59:00.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:59:00.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:59:00.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:59:00.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:59:00.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:59:00.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:59:00.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:59:00.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:59:00.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:59:00.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:59:00.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:59:00.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:59:00.704 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:59:00.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3ae1a832-ab51-45a4-92a1-c129a1b4b389 2021-07-26T06:59:00.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3ae1a832-ab51-45a4-92a1-c129a1b4b389' 2021-07-26T06:59:00.705 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 3ae1a832-ab51-45a4-92a1-c129a1b4b389 2021-07-26T06:59:00.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:59:00.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA0Xf5ge8UiKhAAyVnUykZtcJfmM+1IFmdjtA== 2021-07-26T06:59:00.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA0Xf5ge8UiKhAAyVnUykZtcJfmM+1IFmdjtA=="}' 2021-07-26T06:59:00.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3ae1a832-ab51-45a4-92a1-c129a1b4b389 -i td/osd-scrub-repair/1/new.json 2021-07-26T06:59:01.121 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:59:01.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T06:59:01.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA0Xf5ge8UiKhAAyVnUykZtcJfmM+1IFmdjtA== --osd-uuid 3ae1a832-ab51-45a4-92a1-c129a1b4b389 2021-07-26T06:59:01.159 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:01.157+0000 7f5bc7ec70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:01.159 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:01.158+0000 7f5bc7ec70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:01.162 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:01.159+0000 7f5bc7ec70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:01.162 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:01.160+0000 7f5bc7ec70c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:59:01.162 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:01.160+0000 7f5bc7ec70c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:59:01.163 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:01.160+0000 7f5bc7ec70c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T06:59:01.230 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:01.229+0000 7f5bc7ec70c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T06:59:06.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T06:59:06.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:59:06.681 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T06:59:06.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:59:06.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:59:07.066 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T06:59:07.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:59:07.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:59:07.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:59:07.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:59:07.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:59:07.088 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:07.087+0000 7f9c40a3a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:07.089 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:07.089+0000 7f9c40a3a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:07.090 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:07.090+0000 7f9c40a3a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:07.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:59:07.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:59:07.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:59:07.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:59:07.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:59:07.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:07.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:59:07.339 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:59:07.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:07.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:59:07.610 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:07.610+0000 7f9c40a3a0c0 -1 Falling back to public interface 2021-07-26T06:59:07.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:59:08.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:59:08.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:08.758 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:59:08.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:59:08.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:08.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:59:09.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:59:09.931 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:09.931+0000 7f9c40a3a0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:59:10.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:59:10.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:10.103 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:59:10.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:59:10.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:10.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:59:10.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:59:11.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:59:11.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:11.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:59:11.463 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:59:11.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:11.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:59:11.738 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/315277,v1:127.0.0.1:6811/315277] [v2:127.0.0.1:6812/315277,v1:127.0.0.1:6813/315277] exists,up 3ae1a832-ab51-45a4-92a1-c129a1b4b389 2021-07-26T06:59:11.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:59:11.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:59:11.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:59:11.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: for id in $(seq 0 2) 2021-07-26T06:59:11.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3617: corrupt_scrub_erasure: '[' true = true ']' 2021-07-26T06:59:11.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2021-07-26T06:59:11.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T06:59:11.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:59:11.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:59:11.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:59:11.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T06:59:11.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:59:11.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:59:11.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:59:11.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:59:11.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T06:59:11.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T06:59:11.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:59:11.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:59:11.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:59:11.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:59:11.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:59:11.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:59:11.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:59:11.756 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:59:11.756 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:59:11.756 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:59:11.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:59:11.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:59:11.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:59:11.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:59:11.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:59:11.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:59:11.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:59:11.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:59:11.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:59:11.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:59:11.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:59:11.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:59:11.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T06:59:11.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:59:11.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d082b4f6-4b4a-4e21-9d2c-71764c08edc4 2021-07-26T06:59:11.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d082b4f6-4b4a-4e21-9d2c-71764c08edc4' 2021-07-26T06:59:11.762 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 d082b4f6-4b4a-4e21-9d2c-71764c08edc4 2021-07-26T06:59:11.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:59:11.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/Xf5gRy4NLRAA4bN34hOnA20JDpjOWGtyvg== 2021-07-26T06:59:11.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/Xf5gRy4NLRAA4bN34hOnA20JDpjOWGtyvg=="}' 2021-07-26T06:59:11.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d082b4f6-4b4a-4e21-9d2c-71764c08edc4 -i td/osd-scrub-repair/2/new.json 2021-07-26T06:59:12.134 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:59:12.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2021-07-26T06:59:12.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA/Xf5gRy4NLRAA4bN34hOnA20JDpjOWGtyvg== --osd-uuid d082b4f6-4b4a-4e21-9d2c-71764c08edc4 2021-07-26T06:59:12.172 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:12.170+0000 7f1de15330c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:12.192 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:12.192+0000 7f1de15330c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:12.194 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:12.193+0000 7f1de15330c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:12.194 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:12.193+0000 7f1de15330c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:59:12.194 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:12.193+0000 7f1de15330c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:59:12.194 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:12.193+0000 7f1de15330c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T06:59:12.272 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:12.268+0000 7f1de15330c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2021-07-26T06:59:18.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T06:59:18.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:59:18.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:59:18.011 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T06:59:18.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:59:18.353 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T06:59:18.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:59:18.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:59:18.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:59:18.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:59:18.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:59:18.375 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:18.374+0000 7eff172d10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:18.376 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:18.376+0000 7eff172d10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:18.377 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:18.377+0000 7eff172d10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:18.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:59:18.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:59:18.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:59:18.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:59:18.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:59:18.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:18.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:59:18.663 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T06:59:18.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:18.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:59:18.893 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:18.892+0000 7eff172d10c0 -1 Falling back to public interface 2021-07-26T06:59:18.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:59:19.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:59:19.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:19.970 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T06:59:19.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:59:19.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:19.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:59:20.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:59:21.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:59:21.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:21.247 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T06:59:21.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:59:21.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:21.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:59:21.324 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:21.324+0000 7eff172d10c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:59:21.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:59:22.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:59:22.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:22.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:59:22.550 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T06:59:22.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:22.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:59:22.825 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/316684,v1:127.0.0.1:6819/316684] [v2:127.0.0.1:6820/316684,v1:127.0.0.1:6821/316684] exists,up d082b4f6-4b4a-4e21-9d2c-71764c08edc4 2021-07-26T06:59:22.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:59:22.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:59:22.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:59:22.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: create_rbd_pool 2021-07-26T06:59:22.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:59:23.117 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T06:59:23.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:59:23.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:59:23.556 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T06:59:23.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:59:24.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:59:26.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3624: corrupt_scrub_erasure: create_pool foo 1 2021-07-26T06:59:26.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2021-07-26T06:59:26.906 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'foo' created 2021-07-26T06:59:26.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:59:27.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3626: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2021-07-26T06:59:27.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=ecpool 2021-07-26T06:59:27.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T06:59:27.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=true 2021-07-26T06:59:27.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T06:59:27.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2021-07-26T06:59:28.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2021-07-26T06:59:28.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2021-07-26T06:59:29.167 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' created 2021-07-26T06:59:29.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:59:30.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' true = true ']' 2021-07-26T06:59:30.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2021-07-26T06:59:30.700 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 3 allow_ec_overwrites to true 2021-07-26T06:59:30.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T06:59:30.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:59:30.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:59:30.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:59:30.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:59:30.711 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:59:30.711 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:59:30.712 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:59:30.712 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:59:30.712 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:59:30.712 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:59:30.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:59:30.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:59:30.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:59:30.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:59:30.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:59:31.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:59:31.072 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:59:31.072 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:59:31.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:59:31.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:59:31.073 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:59:31.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836489 2021-07-26T06:59:31.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836489 2021-07-26T06:59:31.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489' 2021-07-26T06:59:31.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:59:31.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:59:31.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672966 2021-07-26T06:59:31.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672966 2021-07-26T06:59:31.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672966' 2021-07-26T06:59:31.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:59:31.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:59:31.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509443 2021-07-26T06:59:31.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509443 2021-07-26T06:59:31.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509443' 2021-07-26T06:59:31.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:59:31.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836489 2021-07-26T06:59:31.432 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:59:31.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:59:31.433 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836489 2021-07-26T06:59:31.433 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:59:31.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836489 2021-07-26T06:59:31.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2021-07-26T06:59:31.435 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836489 2021-07-26T06:59:31.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:59:31.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836489 2021-07-26T06:59:31.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:59:32.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:59:32.699 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:59:32.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836489 2021-07-26T06:59:32.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:59:32.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672966 2021-07-26T06:59:32.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:59:32.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:59:32.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672966 2021-07-26T06:59:32.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:59:32.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672966 2021-07-26T06:59:32.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-07-26T06:59:32.980 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672966 2021-07-26T06:59:32.980 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:59:33.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672967 -lt 42949672966 2021-07-26T06:59:33.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:59:33.246 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509443 2021-07-26T06:59:33.246 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:59:33.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:59:33.247 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509443 2021-07-26T06:59:33.247 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:59:33.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509443 2021-07-26T06:59:33.248 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2021-07-26T06:59:33.249 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509443 2021-07-26T06:59:33.249 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:59:33.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509443 -lt 64424509443 2021-07-26T06:59:33.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:59:33.569 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:33.569 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:33.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:59:33.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:33.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:33.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:33.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:33.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:33.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:33.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:34.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:59:34.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:34.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:34.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:34.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:59:34.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:59:34.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:59:34.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T06:59:34.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3627: corrupt_scrub_erasure: wait_for_clean 2021-07-26T06:59:34.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:59:34.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:59:34.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:59:34.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:59:34.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:59:34.579 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:59:34.579 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:59:34.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:59:34.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:59:34.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:59:34.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:59:34.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:59:34.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:59:34.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:59:34.673 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:59:34.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:59:35.038 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:59:35.038 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:59:35.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:59:35.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:59:35.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:59:35.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836490 2021-07-26T06:59:35.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836490 2021-07-26T06:59:35.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836490' 2021-07-26T06:59:35.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:59:35.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:59:35.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672968 2021-07-26T06:59:35.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672968 2021-07-26T06:59:35.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2021-07-26T06:59:35.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:59:35.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:59:35.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509445 2021-07-26T06:59:35.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509445 2021-07-26T06:59:35.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509445' 2021-07-26T06:59:35.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:59:35.332 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836490 2021-07-26T06:59:35.333 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:59:35.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:59:35.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836490 2021-07-26T06:59:35.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:59:35.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836490 2021-07-26T06:59:35.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-07-26T06:59:35.336 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836490 2021-07-26T06:59:35.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:59:35.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836490 2021-07-26T06:59:35.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:59:36.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:59:36.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:59:36.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836491 -lt 21474836490 2021-07-26T06:59:36.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:59:36.878 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672968 2021-07-26T06:59:36.878 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:59:36.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:59:36.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672968 2021-07-26T06:59:36.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:59:36.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672968 2021-07-26T06:59:36.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-07-26T06:59:36.881 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672968 2021-07-26T06:59:36.882 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:59:37.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672969 -lt 42949672968 2021-07-26T06:59:37.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:59:37.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509445 2021-07-26T06:59:37.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:59:37.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:59:37.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509445 2021-07-26T06:59:37.206 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:59:37.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509445 2021-07-26T06:59:37.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2021-07-26T06:59:37.207 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509445 2021-07-26T06:59:37.207 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:59:37.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509445 -lt 64424509445 2021-07-26T06:59:37.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:59:37.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:37.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:37.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:59:37.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:37.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:37.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:37.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:37.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:37.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:37.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:38.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:59:38.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:38.159 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:38.159 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:38.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:59:38.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:59:38.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:59:38.485 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: seq 1 7 2021-07-26T06:59:38.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T06:59:38.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ1 2021-07-26T06:59:38.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2021-07-26T06:59:38.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:59:38.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T06:59:38.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ1 2021-07-26T06:59:38.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:59:38.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:59:38.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:59:38.915 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:59:38.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:59:39.450 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:59:39.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:59:39.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:59:39.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:59:39.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 1 % 2 2021-07-26T06:59:39.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=1 2021-07-26T06:59:39.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T06:59:39.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: local payload=UVWXYZZZ 2021-07-26T06:59:39.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: echo UVWXYZZZ 2021-07-26T06:59:39.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3640: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:59:39.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:59:39.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:59:39.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T06:59:39.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:59:39.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:59:39.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:59:39.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:59:39.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T06:59:39.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:59:39.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T06:59:39.699 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:59:39.699 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:59:39.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:59:39.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:59:39.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:59:39.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:59:39.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:59:39.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:59:39.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:59:39.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:59:39.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:59:39.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:59:39.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:59:41.505 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:59:41.506 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:59:41.506 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:59:41.506 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s0_head,0#3:00000000::::head#, (61) No data available 2021-07-26T06:59:41.506 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:59:41.507 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T06:59:42.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T06:59:42.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:59:42.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:59:42.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:59:42.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:59:42.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T06:59:42.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:59:42.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:59:42.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:59:42.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:59:42.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T06:59:42.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T06:59:42.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:59:42.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:59:42.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:59:42.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:59:42.078 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:59:42.078 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:59:42.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:59:42.079 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:59:42.079 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:59:42.079 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:59:42.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:59:42.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:59:42.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:59:42.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:59:42.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:59:42.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:59:42.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:59:42.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:59:42.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:59:42.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:59:42.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T06:59:42.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:59:42.082 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T06:59:42.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:59:42.083 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T06:59:42.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:59:42.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:59:42.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:59:42.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:59:42.097 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:42.095+0000 7f64333800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:42.099 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:42.097+0000 7f64333800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:42.100 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:42.099+0000 7f64333800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:42.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:59:42.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:59:42.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:59:42.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:59:42.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:59:42.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:42.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:59:42.356 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:59:42.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:42.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:59:42.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:59:42.623 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:42.622+0000 7f64333800c0 -1 Falling back to public interface 2021-07-26T06:59:43.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:59:43.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:43.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:59:43.622 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:59:43.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:43.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:59:43.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:59:44.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:59:44.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:44.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:59:44.900 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T06:59:44.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:44.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:59:45.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:59:45.319 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:45.317+0000 7f64333800c0 -1 osd.1 36 log_to_monitors {default=true} 2021-07-26T06:59:46.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:59:46.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:46.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:59:46.180 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T06:59:46.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:46.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:59:46.538 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/318647,v1:127.0.0.1:6811/318647] [v2:127.0.0.1:6812/318647,v1:127.0.0.1:6813/318647] exists,up 3ae1a832-ab51-45a4-92a1-c129a1b4b389 2021-07-26T06:59:46.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:59:46.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:59:46.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:59:46.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T06:59:46.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:59:46.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:59:46.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:59:46.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:59:46.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:59:46.541 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:59:46.541 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:59:46.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:59:46.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:59:46.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:59:46.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:59:46.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:59:46.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:59:46.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:59:46.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:59:46.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:59:46.904 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T06:59:46.905 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T06:59:46.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:59:46.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:59:46.905 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:59:47.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836494 2021-07-26T06:59:47.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836494 2021-07-26T06:59:47.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494' 2021-07-26T06:59:47.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:59:47.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:59:47.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691842 2021-07-26T06:59:47.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691842 2021-07-26T06:59:47.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-171798691842' 2021-07-26T06:59:47.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:59:47.143 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:59:47.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509448 2021-07-26T06:59:47.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509448 2021-07-26T06:59:47.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-171798691842 2-64424509448' 2021-07-26T06:59:47.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:59:47.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836494 2021-07-26T06:59:47.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:59:47.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:59:47.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836494 2021-07-26T06:59:47.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:59:47.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836494 2021-07-26T06:59:47.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2021-07-26T06:59:47.265 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836494 2021-07-26T06:59:47.265 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:59:47.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836493 -lt 21474836494 2021-07-26T06:59:47.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:59:48.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:59:48.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:59:48.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836494 -lt 21474836494 2021-07-26T06:59:48.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:59:48.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-171798691842 2021-07-26T06:59:48.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:59:48.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:59:48.811 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-171798691842 2021-07-26T06:59:48.811 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:59:48.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691842 2021-07-26T06:59:48.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2021-07-26T06:59:48.812 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 171798691842 2021-07-26T06:59:48.812 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:59:49.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691842 -lt 171798691842 2021-07-26T06:59:49.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:59:49.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509448 2021-07-26T06:59:49.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:59:49.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:59:49.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509448 2021-07-26T06:59:49.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:59:49.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509448 2021-07-26T06:59:49.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2021-07-26T06:59:49.085 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509448 2021-07-26T06:59:49.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:59:49.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509449 -lt 64424509448 2021-07-26T06:59:49.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:59:49.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:49.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:49.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T06:59:49.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:49.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:49.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:49.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:49.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:49.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:49.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:50.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T06:59:50.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:50.072 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:50.072 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:50.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T06:59:50.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:59:50.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:59:50.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T06:59:50.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ2 2021-07-26T06:59:50.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2021-07-26T06:59:50.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T06:59:50.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T06:59:50.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ2 2021-07-26T06:59:50.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T06:59:50.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T06:59:50.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T06:59:50.912 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T06:59:50.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T06:59:51.355 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T06:59:51.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T06:59:51.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T06:59:51.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2021-07-26T06:59:56.659 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 2 % 2 2021-07-26T06:59:56.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=0 2021-07-26T06:59:56.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T06:59:56.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2021-07-26T06:59:56.663 INFO:tasks.workunit.client.0.smithi071.stderr:1+0 records in 2021-07-26T06:59:56.663 INFO:tasks.workunit.client.0.smithi071.stderr:1+0 records out 2021-07-26T06:59:56.663 INFO:tasks.workunit.client.0.smithi071.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 8.3227e-05 s, 24.6 MB/s 2021-07-26T06:59:56.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:59:56.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T06:59:56.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T06:59:56.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T06:59:56.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T06:59:56.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:59:56.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T06:59:56.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T06:59:56.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T06:59:56.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T06:59:56.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T06:59:56.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:59:56.667 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:59:56.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:59:56.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:59:56.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:59:56.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:59:56.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:59:56.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T06:59:56.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:59:56.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:59:56.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:59:56.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:59:56.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T06:59:58.567 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T06:59:58.568 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T06:59:58.568 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T06:59:58.569 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T06:59:58.572 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T06:59:58.572 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s2_head,2#3:00000000::::head#, (61) No data available 2021-07-26T06:59:59.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T06:59:59.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T06:59:59.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:59:59.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:59:59.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:59:59.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T06:59:59.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T06:59:59.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:59:59.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:59:59.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:59:59.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T06:59:59.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T06:59:59.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:59:59.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:59:59.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:59:59.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T06:59:59.171 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:59:59.171 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:59:59.171 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:59:59.172 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:59:59.172 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:59:59.172 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T06:59:59.172 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:59:59.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T06:59:59.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:59:59.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T06:59:59.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T06:59:59.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:59:59.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:59:59.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:59:59.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:59:59.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:59:59.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T06:59:59.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:59:59.175 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T06:59:59.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:59:59.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T06:59:59.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:59:59.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:59:59.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:59:59.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:59:59.192 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:59.190+0000 7fe6fede70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:59.194 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:59.193+0000 7fe6fede70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:59.195 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:59.194+0000 7fe6fede70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:59:59.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:59:59.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:59:59.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:59:59.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:59:59.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:59:59.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:59:59.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:59:59.456 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T06:59:59.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:59:59.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:59:59.713 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T06:59:59.711+0000 7fe6fede70c0 -1 Falling back to public interface 2021-07-26T06:59:59.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:00.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:00.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:00.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:00:00.769 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:00:00.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:00.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:00:01.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:02.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:02.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:02.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:00:02.077 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:00:02.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:02.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:00:02.156 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:02.156+0000 7fe6fede70c0 -1 osd.0 43 log_to_monitors {default=true} 2021-07-26T07:00:02.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:03.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:03.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:03.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:00:03.415 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:00:03.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:03.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:00:03.767 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 47 up_thru 47 down_at 44 last_clean_interval [5,43) [v2:127.0.0.1:6802/320064,v1:127.0.0.1:6803/320064] [v2:127.0.0.1:6804/320064,v1:127.0.0.1:6805/320064] exists,up c3ed524e-cfd1-4ff6-b405-a9438f239b1a 2021-07-26T07:00:03.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:00:03.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:00:03.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:00:03.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:00:03.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:00:03.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:00:03.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:00:03.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:00:03.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:00:03.769 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:00:03.770 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:00:03.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:00:03.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:00:03.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:00:03.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:00:03.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:00:03.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:00:03.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:00:03.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:00:04.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:00:04.145 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:00:04.146 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:00:04.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:00:04.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:04.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:00:04.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=201863462914 2021-07-26T07:00:04.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 201863462914 2021-07-26T07:00:04.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-201863462914' 2021-07-26T07:00:04.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:04.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:00:04.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691846 2021-07-26T07:00:04.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691846 2021-07-26T07:00:04.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-201863462914 1-171798691846' 2021-07-26T07:00:04.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:04.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:00:04.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509453 2021-07-26T07:00:04.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509453 2021-07-26T07:00:04.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-201863462914 1-171798691846 2-64424509453' 2021-07-26T07:00:04.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:04.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-201863462914 2021-07-26T07:00:04.667 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:04.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:00:04.668 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-201863462914 2021-07-26T07:00:04.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:04.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=201863462914 2021-07-26T07:00:04.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 201863462914' 2021-07-26T07:00:04.669 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 201863462914 2021-07-26T07:00:04.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:00:04.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 201863462914 -lt 201863462914 2021-07-26T07:00:04.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:04.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-171798691846 2021-07-26T07:00:04.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:04.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:00:04.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-171798691846 2021-07-26T07:00:04.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:04.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691846 2021-07-26T07:00:04.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 171798691846' 2021-07-26T07:00:04.942 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 171798691846 2021-07-26T07:00:04.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:00:05.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691846 -lt 171798691846 2021-07-26T07:00:05.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:05.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509453 2021-07-26T07:00:05.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:05.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:00:05.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509453 2021-07-26T07:00:05.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:05.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509453 2021-07-26T07:00:05.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2021-07-26T07:00:05.246 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509453 2021-07-26T07:00:05.246 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:00:05.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509452 -lt 64424509453 2021-07-26T07:00:05.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:00:06.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:00:06.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:00:06.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509453 -lt 64424509453 2021-07-26T07:00:06.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:00:06.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:06.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:07.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:00:07.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:07.151 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:07.151 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:07.151 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:07.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:07.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:07.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:07.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:00:07.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:07.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:07.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:07.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:00:07.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:00:07.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:00:07.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T07:00:07.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ3 2021-07-26T07:00:07.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2021-07-26T07:00:07.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:00:07.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T07:00:07.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ3 2021-07-26T07:00:07.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:00:07.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:00:07.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:00:08.322 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:00:08.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:00:08.723 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:00:08.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:00:08.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:00:08.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:00:08.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 3 % 2 2021-07-26T07:00:08.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=1 2021-07-26T07:00:08.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T07:00:08.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3651: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2021-07-26T07:00:08.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:00:08.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:00:08.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:00:08.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:00:08.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2021-07-26T07:00:08.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:00:08.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:00:08.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:00:08.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:00:08.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:00:08.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:00:08.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:00:08.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:00:08.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:00:08.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:00:09.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:00:09.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2021-07-26T07:00:09.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:00:09.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:00:09.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:00:09.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:00:09.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:00:09.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2021-07-26T07:00:10.960 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:00:10.961 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:00:10.961 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:00:10.961 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s0_head,0#3:00000000::::head#, (61) No data available 2021-07-26T07:00:10.961 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:00:10.962 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:00:10.962 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2021-07-26T07:00:11.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:00:11.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:00:11.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:00:11.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:00:11.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:00:11.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:00:11.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:00:11.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:00:11.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:00:11.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:00:11.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:00:11.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:00:11.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:00:11.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:00:11.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:00:11.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:00:11.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:00:11.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:00:11.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:00:11.488 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:00:11.488 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:00:11.488 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:00:11.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:00:11.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:00:11.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:00:11.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:00:11.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:00:11.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:00:11.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:00:11.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:00:11.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:00:11.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:00:11.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:00:11.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:00:11.492 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:00:11.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:00:11.493 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:00:11.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:00:11.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:00:11.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:00:11.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:00:11.505 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:11.505+0000 7fb705c8a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:00:11.507 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:11.507+0000 7fb705c8a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:00:11.508 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:11.508+0000 7fb705c8a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:00:11.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:00:11.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:00:11.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:00:11.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:00:11.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:00:11.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:11.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:00:11.762 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:00:11.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:11.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:00:12.024 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:12.024+0000 7fb705c8a0c0 -1 Falling back to public interface 2021-07-26T07:00:12.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:13.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:13.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:13.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:00:13.072 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:00:13.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:13.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:00:13.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:14.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:14.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:14.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:00:14.347 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:00:14.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:14.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:00:14.454 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:14.453+0000 7fb705c8a0c0 -1 osd.1 51 log_to_monitors {default=true} 2021-07-26T07:00:14.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:15.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:15.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:15.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:00:15.664 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:00:15.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:15.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:00:15.968 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 52 last_clean_interval [40,51) [v2:127.0.0.1:6810/321486,v1:127.0.0.1:6811/321486] [v2:127.0.0.1:6812/321486,v1:127.0.0.1:6813/321486] exists,up 3ae1a832-ab51-45a4-92a1-c129a1b4b389 2021-07-26T07:00:15.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:00:15.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:00:15.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:00:15.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:00:15.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:00:15.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:00:15.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:00:15.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:00:15.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:00:15.971 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:00:15.971 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:00:15.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:00:15.972 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:00:15.972 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:00:16.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:00:16.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:00:16.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:00:16.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:00:16.065 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:00:16.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:00:16.378 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:00:16.378 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:00:16.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:00:16.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:16.379 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:00:16.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=201863462917 2021-07-26T07:00:16.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 201863462917 2021-07-26T07:00:16.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-201863462917' 2021-07-26T07:00:16.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:16.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:00:16.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=240518168578 2021-07-26T07:00:16.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 240518168578 2021-07-26T07:00:16.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-201863462917 1-240518168578' 2021-07-26T07:00:16.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:16.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:00:16.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509456 2021-07-26T07:00:16.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509456 2021-07-26T07:00:16.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-201863462917 1-240518168578 2-64424509456' 2021-07-26T07:00:16.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:16.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-201863462917 2021-07-26T07:00:16.744 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:16.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:00:16.746 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-201863462917 2021-07-26T07:00:16.746 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:16.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=201863462917 2021-07-26T07:00:16.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 201863462917' 2021-07-26T07:00:16.747 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 201863462917 2021-07-26T07:00:16.747 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:00:17.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 201863462917 -lt 201863462917 2021-07-26T07:00:17.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:17.016 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-240518168578 2021-07-26T07:00:17.016 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:17.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:00:17.017 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-240518168578 2021-07-26T07:00:17.017 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:17.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=240518168578 2021-07-26T07:00:17.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 240518168578' 2021-07-26T07:00:17.019 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 240518168578 2021-07-26T07:00:17.019 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:00:17.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 240518168578 2021-07-26T07:00:17.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:00:18.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:00:18.287 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:00:18.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 240518168578 2021-07-26T07:00:18.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:00:19.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:00:19.567 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:00:19.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 240518168578 -lt 240518168578 2021-07-26T07:00:19.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:19.849 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509456 2021-07-26T07:00:19.849 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:19.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:00:19.850 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509456 2021-07-26T07:00:19.851 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:19.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509456 2021-07-26T07:00:19.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2021-07-26T07:00:19.852 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509456 2021-07-26T07:00:19.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:00:20.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509457 -lt 64424509456 2021-07-26T07:00:20.159 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:00:20.161 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:20.161 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:20.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:00:20.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:20.525 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:20.526 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:20.526 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:20.526 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:20.526 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:20.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:20.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:00:20.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:20.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:20.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:21.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:00:21.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:00:21.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:00:21.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T07:00:21.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ4 2021-07-26T07:00:21.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2021-07-26T07:00:21.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:00:21.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T07:00:21.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ4 2021-07-26T07:00:21.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:00:21.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:00:21.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:00:21.621 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:00:21.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:00:22.123 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:00:22.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:00:22.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:00:22.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:00:26.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 4 % 2 2021-07-26T07:00:26.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=0 2021-07-26T07:00:26.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T07:00:26.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3655: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2021-07-26T07:00:26.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3656: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2021-07-26T07:00:26.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: echo -n bad-val 2021-07-26T07:00:26.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3660: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2021-07-26T07:00:26.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:00:26.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:00:26.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:00:26.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:00:26.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2021-07-26T07:00:26.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:00:26.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:00:26.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:00:26.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:00:26.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:00:26.917 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:00:26.917 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:00:26.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:00:26.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:00:26.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:00:26.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:00:26.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2021-07-26T07:00:26.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:00:26.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:00:26.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:00:26.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:00:26.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:00:26.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2021-07-26T07:00:28.390 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:00:28.390 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:00:28.390 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:00:28.391 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:00:28.391 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:00:28.391 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s2_head,2#3:00000000::::head#, (61) No data available 2021-07-26T07:00:28.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:00:28.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:00:28.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:00:28.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:00:28.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:00:28.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:00:28.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:00:28.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:00:28.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:00:28.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:00:28.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:00:28.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:00:28.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:00:28.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:00:28.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:00:28.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:00:28.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:00:28.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:00:28.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:00:28.914 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:00:28.914 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:00:28.915 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:00:28.915 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:00:28.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:00:28.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:00:28.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:00:28.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:00:28.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:00:28.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:00:28.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:00:28.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:00:28.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:00:28.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:00:28.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:00:28.924 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:00:28.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:00:28.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:00:28.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:00:28.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:00:28.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:00:28.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:00:28.934 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:28.934+0000 7efef74d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:00:28.936 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:28.936+0000 7efef74d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:00:28.937 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:28.937+0000 7efef74d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:00:29.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:00:29.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:00:29.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:00:29.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:00:29.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:00:29.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:29.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:00:29.193 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:00:29.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:29.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:00:29.453 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:29.453+0000 7efef74d80c0 -1 Falling back to public interface 2021-07-26T07:00:29.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:30.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:30.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:30.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:00:30.497 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:00:30.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:30.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:00:30.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:31.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:31.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:31.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:00:31.789 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:00:31.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:31.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:00:31.945 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:31.945+0000 7efef74d80c0 -1 osd.0 60 log_to_monitors {default=true} 2021-07-26T07:00:32.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:33.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:33.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:33.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:00:33.060 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:00:33.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:33.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:00:33.443 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 65 up_thru 65 down_at 61 last_clean_interval [47,60) [v2:127.0.0.1:6802/322973,v1:127.0.0.1:6803/322973] [v2:127.0.0.1:6804/322973,v1:127.0.0.1:6805/322973] exists,up c3ed524e-cfd1-4ff6-b405-a9438f239b1a 2021-07-26T07:00:33.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:00:33.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:00:33.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:00:33.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:00:33.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:00:33.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:00:33.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:00:33.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:00:33.446 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:00:33.446 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:00:33.446 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:00:33.446 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:00:33.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:00:33.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:00:33.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:00:33.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:00:33.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:00:33.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:00:33.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:00:33.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:00:33.905 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:00:33.906 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:00:33.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:00:33.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:33.906 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:00:34.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=279172874242 2021-07-26T07:00:34.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 279172874242 2021-07-26T07:00:34.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-279172874242' 2021-07-26T07:00:34.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:34.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:00:34.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=240518168582 2021-07-26T07:00:34.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 240518168582 2021-07-26T07:00:34.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-279172874242 1-240518168582' 2021-07-26T07:00:34.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:34.142 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:00:34.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509461 2021-07-26T07:00:34.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509461 2021-07-26T07:00:34.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-279172874242 1-240518168582 2-64424509461' 2021-07-26T07:00:34.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:34.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-279172874242 2021-07-26T07:00:34.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:34.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:00:34.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-279172874242 2021-07-26T07:00:34.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:34.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=279172874242 2021-07-26T07:00:34.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 279172874242' 2021-07-26T07:00:34.263 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 279172874242 2021-07-26T07:00:34.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:00:34.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 279172874242 2021-07-26T07:00:34.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:00:35.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:00:35.530 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:00:35.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 279172874242 -lt 279172874242 2021-07-26T07:00:35.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:35.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-240518168582 2021-07-26T07:00:35.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:35.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:00:35.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-240518168582 2021-07-26T07:00:35.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:35.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=240518168582 2021-07-26T07:00:35.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 240518168582' 2021-07-26T07:00:35.807 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 240518168582 2021-07-26T07:00:35.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:00:36.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 240518168582 -lt 240518168582 2021-07-26T07:00:36.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:36.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509461 2021-07-26T07:00:36.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:36.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:00:36.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509461 2021-07-26T07:00:36.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:36.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509461 2021-07-26T07:00:36.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2021-07-26T07:00:36.109 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 64424509461 2021-07-26T07:00:36.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:00:36.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509461 -lt 64424509461 2021-07-26T07:00:36.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:00:36.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:36.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:36.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:00:36.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:36.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:36.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:36.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:36.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:36.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:36.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:37.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:00:37.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:37.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:37.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:37.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:00:37.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:00:37.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:00:37.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3661: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2021-07-26T07:00:37.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:00:37.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:00:37.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=2 2021-07-26T07:00:37.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:00:37.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2021-07-26T07:00:37.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:00:37.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:00:37.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=2 2021-07-26T07:00:37.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:00:37.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2021-07-26T07:00:37.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:00:37.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:00:37.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:00:37.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:00:37.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:00:37.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:00:37.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2021-07-26T07:00:37.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:00:37.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:00:37.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=2 2021-07-26T07:00:37.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:00:37.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:00:37.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2021-07-26T07:00:39.409 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:00:39.410 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:00:39.410 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:00:39.410 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:00:39.411 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:00:39.411 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s1_head,1#3:00000000::::head#, (61) No data available 2021-07-26T07:00:39.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2021-07-26T07:00:39.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:00:39.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:00:39.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T07:00:39.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:00:39.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:00:39.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:00:39.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:00:39.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:00:39.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:00:39.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T07:00:39.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T07:00:39.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:00:39.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:00:39.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:00:39.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:00:39.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:00:39.938 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:00:39.938 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:00:39.938 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:00:39.938 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:00:39.939 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:00:39.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:00:39.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:00:39.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:00:39.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:00:39.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:00:39.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:00:39.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:00:39.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:00:39.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:00:39.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:00:39.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T07:00:39.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T07:00:39.941 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.2 2021-07-26T07:00:39.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:00:39.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/2/whoami 2021-07-26T07:00:39.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T07:00:39.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:00:39.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:00:39.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:00:39.957 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:39.956+0000 7ffb8fc7e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:00:39.959 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:39.958+0000 7ffb8fc7e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:00:39.960 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:39.959+0000 7ffb8fc7e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:00:40.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T07:00:40.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:00:40.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:00:40.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:00:40.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:00:40.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:40.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:00:40.221 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:00:40.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:40.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:00:40.480 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:40.479+0000 7ffb8fc7e0c0 -1 Falling back to public interface 2021-07-26T07:00:40.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:41.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:41.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:41.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:00:41.520 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:00:41.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:41.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:00:41.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:42.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:42.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:42.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:00:42.798 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:00:42.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:42.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:00:42.918 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:42.916+0000 7ffb8fc7e0c0 -1 osd.2 67 log_to_monitors {default=true} 2021-07-26T07:00:43.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:44.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:44.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:44.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:00:44.178 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:00:44.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:44.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:00:44.486 INFO:tasks.workunit.client.0.smithi071.stderr:osd.2 up in weight 1 up_from 72 up_thru 72 down_at 68 last_clean_interval [15,67) [v2:127.0.0.1:6818/324321,v1:127.0.0.1:6819/324321] [v2:127.0.0.1:6820/324321,v1:127.0.0.1:6821/324321] exists,up d082b4f6-4b4a-4e21-9d2c-71764c08edc4 2021-07-26T07:00:44.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:00:44.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:00:44.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:00:44.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:00:44.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:00:44.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:00:44.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:00:44.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:00:44.488 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:00:44.489 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:00:44.489 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:00:44.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:00:44.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:00:44.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:00:44.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:00:44.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:00:44.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:00:44.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:00:44.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:00:44.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:00:44.971 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:00:44.971 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:00:44.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:00:44.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:44.972 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:00:45.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=279172874245 2021-07-26T07:00:45.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 279172874245 2021-07-26T07:00:45.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-279172874245' 2021-07-26T07:00:45.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:45.090 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:00:45.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=240518168586 2021-07-26T07:00:45.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 240518168586 2021-07-26T07:00:45.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-279172874245 1-240518168586' 2021-07-26T07:00:45.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:45.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:00:45.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=309237645314 2021-07-26T07:00:45.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 309237645314 2021-07-26T07:00:45.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-279172874245 1-240518168586 2-309237645314' 2021-07-26T07:00:45.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:45.326 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-279172874245 2021-07-26T07:00:45.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:45.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:00:45.328 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-279172874245 2021-07-26T07:00:45.329 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:45.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=279172874245 2021-07-26T07:00:45.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 279172874245' 2021-07-26T07:00:45.330 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 279172874245 2021-07-26T07:00:45.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:00:45.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 279172874244 -lt 279172874245 2021-07-26T07:00:45.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:00:46.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:00:46.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:00:46.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 279172874245 -lt 279172874245 2021-07-26T07:00:46.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:46.878 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-240518168586 2021-07-26T07:00:46.878 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:46.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:00:46.880 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-240518168586 2021-07-26T07:00:46.880 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:46.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=240518168586 2021-07-26T07:00:46.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 240518168586' 2021-07-26T07:00:46.881 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 240518168586 2021-07-26T07:00:46.881 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:00:47.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 240518168586 -lt 240518168586 2021-07-26T07:00:47.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:47.182 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-309237645314 2021-07-26T07:00:47.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:47.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:00:47.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-309237645314 2021-07-26T07:00:47.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:47.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=309237645314 2021-07-26T07:00:47.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 309237645314' 2021-07-26T07:00:47.185 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 309237645314 2021-07-26T07:00:47.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:00:47.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 309237645314 -lt 309237645314 2021-07-26T07:00:47.483 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:00:47.483 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:47.483 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:47.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:00:47.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:47.849 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:47.850 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:47.850 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:47.850 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:47.850 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:47.850 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:48.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:00:48.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:48.111 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:48.111 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:48.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:00:48.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:00:48.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:00:48.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3662: corrupt_scrub_erasure: echo -n val3-EOBJ4 2021-07-26T07:00:48.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2021-07-26T07:00:48.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:00:48.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:00:48.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=2 2021-07-26T07:00:48.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:00:48.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2021-07-26T07:00:48.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:00:48.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:00:48.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=2 2021-07-26T07:00:48.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:00:48.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2021-07-26T07:00:48.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:00:48.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:00:48.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:00:48.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:00:48.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:00:48.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:00:48.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2021-07-26T07:00:48.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:00:48.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:00:48.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=2 2021-07-26T07:00:48.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:00:48.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:00:48.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2021-07-26T07:00:50.242 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:00:50.242 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:00:50.243 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:00:50.243 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:00:50.243 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:00:50.243 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s1_head,1#3:00000000::::head#, (61) No data available 2021-07-26T07:00:50.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2021-07-26T07:00:50.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:00:50.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:00:50.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T07:00:50.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:00:50.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:00:50.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:00:50.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:00:50.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:00:50.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:00:50.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T07:00:50.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T07:00:50.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:00:50.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:00:50.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:00:50.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:00:50.765 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:00:50.765 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:00:50.765 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:00:50.765 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:00:50.766 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:00:50.766 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:00:50.766 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:00:50.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:00:50.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:00:50.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:00:50.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:00:50.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:00:50.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:00:50.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:00:50.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:00:50.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:00:50.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T07:00:50.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T07:00:50.770 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.2 2021-07-26T07:00:50.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:00:50.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/2/whoami 2021-07-26T07:00:50.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T07:00:50.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:00:50.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:00:50.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:00:50.786 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:50.785+0000 7f7e6f8370c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:00:50.788 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:50.787+0000 7f7e6f8370c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:00:50.789 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:50.788+0000 7f7e6f8370c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:00:51.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T07:00:51.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:00:51.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:00:51.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:00:51.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:00:51.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:51.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:00:51.043 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:00:51.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:51.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:00:51.310 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:51.309+0000 7f7e6f8370c0 -1 Falling back to public interface 2021-07-26T07:00:51.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:52.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:52.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:52.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:00:52.313 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:00:52.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:52.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:00:52.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:53.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:53.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:53.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:00:53.591 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:00:53.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:53.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:00:53.648 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:00:53.646+0000 7f7e6f8370c0 -1 osd.2 74 log_to_monitors {default=true} 2021-07-26T07:00:53.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:00:54.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:00:54.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:00:54.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:00:54.874 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:00:54.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:00:54.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:00:55.268 INFO:tasks.workunit.client.0.smithi071.stderr:osd.2 up in weight 1 up_from 79 up_thru 79 down_at 75 last_clean_interval [72,74) [v2:127.0.0.1:6818/325669,v1:127.0.0.1:6819/325669] [v2:127.0.0.1:6820/325669,v1:127.0.0.1:6821/325669] exists,up d082b4f6-4b4a-4e21-9d2c-71764c08edc4 2021-07-26T07:00:55.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:00:55.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:00:55.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:00:55.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:00:55.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:00:55.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:00:55.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:00:55.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:00:55.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:00:55.270 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:00:55.270 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:00:55.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:00:55.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:00:55.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:00:55.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:00:55.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:00:55.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:00:55.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:00:55.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:00:55.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:00:55.680 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:00:55.680 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:00:55.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:00:55.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:55.681 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:00:55.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=279172874248 2021-07-26T07:00:55.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 279172874248 2021-07-26T07:00:55.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-279172874248' 2021-07-26T07:00:55.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:55.800 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:00:55.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=240518168589 2021-07-26T07:00:55.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 240518168589 2021-07-26T07:00:55.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-279172874248 1-240518168589' 2021-07-26T07:00:55.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:00:55.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:00:56.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=339302416386 2021-07-26T07:00:56.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 339302416386 2021-07-26T07:00:56.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-279172874248 1-240518168589 2-339302416386' 2021-07-26T07:00:56.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:56.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-279172874248 2021-07-26T07:00:56.037 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:56.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:00:56.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-279172874248 2021-07-26T07:00:56.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:56.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=279172874248 2021-07-26T07:00:56.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 279172874248' 2021-07-26T07:00:56.040 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 279172874248 2021-07-26T07:00:56.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:00:56.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 279172874247 -lt 279172874248 2021-07-26T07:00:56.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:00:57.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:00:57.306 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:00:57.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 279172874248 -lt 279172874248 2021-07-26T07:00:57.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:57.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-240518168589 2021-07-26T07:00:57.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:57.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:00:57.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-240518168589 2021-07-26T07:00:57.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:57.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=240518168589 2021-07-26T07:00:57.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 240518168589' 2021-07-26T07:00:57.587 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 240518168589 2021-07-26T07:00:57.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:00:57.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 240518168589 -lt 240518168589 2021-07-26T07:00:57.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:00:57.894 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-339302416386 2021-07-26T07:00:57.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:00:57.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:00:57.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-339302416386 2021-07-26T07:00:57.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:00:57.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=339302416386 2021-07-26T07:00:57.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 339302416386' 2021-07-26T07:00:57.897 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 339302416386 2021-07-26T07:00:57.897 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:00:58.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 339302416386 -lt 339302416386 2021-07-26T07:00:58.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:00:58.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:58.209 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:58.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:00:58.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:58.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:58.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:58.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:58.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:58.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:58.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:58.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:00:58.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:58.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:58.845 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:59.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:00:59.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:00:59.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:00:59.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2021-07-26T07:00:59.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T07:00:59.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ5 2021-07-26T07:00:59.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2021-07-26T07:00:59.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:00:59.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T07:00:59.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ5 2021-07-26T07:00:59.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:00:59.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:00:59.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:00:59.561 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:00:59.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:01:00.063 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:01:00.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:01:00.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:01:00.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:01:00.338 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 5 % 2 2021-07-26T07:01:00.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=1 2021-07-26T07:01:00.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T07:01:00.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2021-07-26T07:01:00.342 INFO:tasks.workunit.client.0.smithi071.stderr:2+0 records in 2021-07-26T07:01:00.343 INFO:tasks.workunit.client.0.smithi071.stderr:2+0 records out 2021-07-26T07:01:00.343 INFO:tasks.workunit.client.0.smithi071.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 7.0657e-05 s, 58.0 MB/s 2021-07-26T07:01:00.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:01:00.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:01:00.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:01:00.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:01:00.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:01:00.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:01:00.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:01:00.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:01:00.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:01:00.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:01:00.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:01:00.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:01:00.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:01:00.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:01:00.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:01:00.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:01:00.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:01:00.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:01:00.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:01:00.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:01:00.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:01:00.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:01:00.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:01:00.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:01:02.198 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:01:02.199 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:01:02.199 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:01:02.199 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s0_head,0#3:00000000::::head#, (61) No data available 2021-07-26T07:01:02.199 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:01:02.200 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:01:02.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:01:02.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:01:02.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:01:02.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:01:02.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:01:02.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:01:02.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:01:02.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:01:02.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:01:02.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:01:02.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:01:02.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:01:02.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:01:02.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:01:02.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:01:02.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:01:02.793 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:01:02.793 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:01:02.793 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:01:02.793 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:01:02.794 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:01:02.794 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:01:02.794 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:01:02.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:01:02.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:01:02.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:01:02.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:01:02.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:01:02.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:01:02.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:01:02.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:01:02.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:01:02.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:01:02.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:01:02.797 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:01:02.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:01:02.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:01:02.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:01:02.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:01:02.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:01:02.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:01:02.814 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:02.814+0000 7fb0721970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:02.816 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:02.816+0000 7fb0721970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:02.817 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:02.817+0000 7fb0721970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:03.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:01:03.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:01:03.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:01:03.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:01:03.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:01:03.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:03.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:01:03.091 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:01:03.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:03.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:01:03.334 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:03.334+0000 7fb0721970c0 -1 Falling back to public interface 2021-07-26T07:01:03.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:01:04.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:01:04.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:04.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:01:04.357 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:01:04.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:04.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:01:04.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:01:05.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:01:05.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:05.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:01:05.636 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:01:05.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:05.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:01:05.762 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:05.762+0000 7fb0721970c0 -1 osd.1 83 log_to_monitors {default=true} 2021-07-26T07:01:06.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:01:07.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:01:07.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:07.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:01:07.014 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:01:07.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:07.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:01:07.402 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 84 last_clean_interval [56,83) [v2:127.0.0.1:6810/327097,v1:127.0.0.1:6811/327097] [v2:127.0.0.1:6812/327097,v1:127.0.0.1:6813/327097] exists,up 3ae1a832-ab51-45a4-92a1-c129a1b4b389 2021-07-26T07:01:07.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:01:07.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:01:07.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:01:07.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:01:07.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:01:07.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:01:07.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:01:07.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:01:07.406 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:01:07.406 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:01:07.407 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:01:07.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:01:07.408 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:01:07.408 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:01:07.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:01:07.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:01:07.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:01:07.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:01:07.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:01:07.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:01:07.837 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:01:07.837 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:01:07.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:01:07.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:01:07.838 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:01:07.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=279172874252 2021-07-26T07:01:07.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 279172874252 2021-07-26T07:01:07.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-279172874252' 2021-07-26T07:01:07.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:01:07.956 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:01:08.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=377957122050 2021-07-26T07:01:08.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 377957122050 2021-07-26T07:01:08.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-279172874252 1-377957122050' 2021-07-26T07:01:08.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:01:08.076 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:01:08.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=339302416389 2021-07-26T07:01:08.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 339302416389 2021-07-26T07:01:08.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-279172874252 1-377957122050 2-339302416389' 2021-07-26T07:01:08.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:01:08.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-279172874252 2021-07-26T07:01:08.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:01:08.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:01:08.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-279172874252 2021-07-26T07:01:08.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:01:08.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=279172874252 2021-07-26T07:01:08.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 279172874252' 2021-07-26T07:01:08.197 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 279172874252 2021-07-26T07:01:08.197 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:01:08.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 279172874250 -lt 279172874252 2021-07-26T07:01:08.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:01:09.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:01:09.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:01:09.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 279172874252 -lt 279172874252 2021-07-26T07:01:09.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:01:09.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-377957122050 2021-07-26T07:01:09.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:01:09.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:01:09.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-377957122050 2021-07-26T07:01:09.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:01:09.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=377957122050 2021-07-26T07:01:09.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2021-07-26T07:01:09.742 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 377957122050 2021-07-26T07:01:09.742 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:01:10.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 377957122050 -lt 377957122050 2021-07-26T07:01:10.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:01:10.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-339302416389 2021-07-26T07:01:10.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:01:10.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:01:10.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-339302416389 2021-07-26T07:01:10.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:01:10.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=339302416389 2021-07-26T07:01:10.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 339302416389' 2021-07-26T07:01:10.041 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 339302416389 2021-07-26T07:01:10.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:01:10.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 339302416389 -lt 339302416389 2021-07-26T07:01:10.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:01:10.323 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:10.323 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:10.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:01:10.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:10.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:10.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:10.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:10.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:10.649 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:10.649 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:10.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:01:10.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:10.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:10.953 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:11.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:01:11.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:01:11.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:01:11.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T07:01:11.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ6 2021-07-26T07:01:11.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2021-07-26T07:01:11.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:01:11.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T07:01:11.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ6 2021-07-26T07:01:11.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:01:11.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:01:11.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:01:11.751 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:01:11.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:01:12.269 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:01:12.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:01:12.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:01:12.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:01:17.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 6 % 2 2021-07-26T07:01:17.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=0 2021-07-26T07:01:17.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T07:01:17.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3674: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2021-07-26T07:01:17.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:01:17.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:01:17.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:01:17.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:01:17.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2021-07-26T07:01:17.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:01:17.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:01:17.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:01:17.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:01:17.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:01:17.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:01:17.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:01:17.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:01:17.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:01:17.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:01:17.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:01:17.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2021-07-26T07:01:17.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:01:17.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:01:17.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:01:17.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:01:17.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:01:17.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2021-07-26T07:01:19.435 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:01:19.435 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:01:19.436 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:01:19.436 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:01:19.436 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:01:19.436 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s2_head,2#3:00000000::::head#, (61) No data available 2021-07-26T07:01:19.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:01:19.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:01:19.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:01:19.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:01:19.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:01:19.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:01:19.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:01:19.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:01:19.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:01:19.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:01:19.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:01:19.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:01:19.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:01:19.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:01:19.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:01:19.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:01:19.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:01:19.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:01:19.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:01:19.962 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:01:19.962 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:01:19.963 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:01:19.963 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:01:19.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:01:19.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:01:19.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:01:19.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:01:19.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:01:19.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:01:19.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:01:19.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:01:19.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:01:19.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:01:19.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:01:19.965 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:01:19.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:01:19.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:01:19.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:01:19.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:01:19.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:01:19.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:01:19.982 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:19.982+0000 7f6c366010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:19.984 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:19.984+0000 7f6c366010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:19.985 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:19.985+0000 7f6c366010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:20.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:01:20.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:01:20.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:01:20.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:01:20.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:01:20.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:20.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:01:20.243 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:01:20.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:20.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:01:20.501 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:20.500+0000 7f6c366010c0 -1 Falling back to public interface 2021-07-26T07:01:20.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:01:21.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:01:21.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:21.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:01:21.510 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:01:21.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:21.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:01:21.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:01:22.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:01:22.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:22.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:01:22.786 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:01:22.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:22.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:01:23.055 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:23.055+0000 7f6c366010c0 -1 osd.0 92 log_to_monitors {default=true} 2021-07-26T07:01:23.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:01:24.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:01:24.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:24.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:01:24.069 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:01:24.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:24.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:01:24.475 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 97 up_thru 97 down_at 93 last_clean_interval [65,92) [v2:127.0.0.1:6802/328520,v1:127.0.0.1:6803/328520] [v2:127.0.0.1:6804/328520,v1:127.0.0.1:6805/328520] exists,up c3ed524e-cfd1-4ff6-b405-a9438f239b1a 2021-07-26T07:01:24.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:01:24.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:01:24.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:01:24.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:01:24.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:01:24.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:01:24.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:01:24.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:01:24.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:01:24.477 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:01:24.477 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:01:24.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:01:24.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:01:24.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:01:24.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:01:24.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:01:24.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:01:24.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:01:24.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:01:24.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:01:24.936 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:01:24.937 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:01:24.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:01:24.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:01:24.938 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:01:25.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=416611827714 2021-07-26T07:01:25.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 416611827714 2021-07-26T07:01:25.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-416611827714' 2021-07-26T07:01:25.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:01:25.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:01:25.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=377957122054 2021-07-26T07:01:25.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 377957122054 2021-07-26T07:01:25.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-416611827714 1-377957122054' 2021-07-26T07:01:25.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:01:25.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:01:25.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=339302416394 2021-07-26T07:01:25.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 339302416394 2021-07-26T07:01:25.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-416611827714 1-377957122054 2-339302416394' 2021-07-26T07:01:25.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:01:25.308 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-416611827714 2021-07-26T07:01:25.308 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:01:25.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:01:25.309 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-416611827714 2021-07-26T07:01:25.309 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:01:25.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=416611827714 2021-07-26T07:01:25.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 416611827714' 2021-07-26T07:01:25.310 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 416611827714 2021-07-26T07:01:25.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:01:25.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 416611827714 2021-07-26T07:01:25.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:01:26.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:01:26.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:01:26.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 416611827714 -lt 416611827714 2021-07-26T07:01:26.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:01:26.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-377957122054 2021-07-26T07:01:26.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:01:26.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:01:26.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-377957122054 2021-07-26T07:01:26.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:01:26.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=377957122054 2021-07-26T07:01:26.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 377957122054' 2021-07-26T07:01:26.855 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 377957122054 2021-07-26T07:01:26.856 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:01:27.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 377957122055 -lt 377957122054 2021-07-26T07:01:27.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:01:27.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-339302416394 2021-07-26T07:01:27.124 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:01:27.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:01:27.125 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-339302416394 2021-07-26T07:01:27.125 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:01:27.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=339302416394 2021-07-26T07:01:27.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 339302416394' 2021-07-26T07:01:27.126 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 339302416394 2021-07-26T07:01:27.126 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:01:27.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 339302416394 -lt 339302416394 2021-07-26T07:01:27.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:01:27.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:27.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:27.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:01:27.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:27.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:27.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:27.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:27.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:27.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:27.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:28.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:01:28.070 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:28.070 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:28.070 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:28.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:01:28.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:01:28.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:01:28.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3675: corrupt_scrub_erasure: echo -n bad-val 2021-07-26T07:01:28.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2021-07-26T07:01:28.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:01:28.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:01:28.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:01:28.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:01:28.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2021-07-26T07:01:28.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:01:28.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:01:28.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:01:28.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:01:28.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:01:28.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:01:28.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:01:28.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:01:28.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:01:28.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:01:28.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:01:28.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2021-07-26T07:01:28.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:01:28.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:01:28.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:01:28.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:01:28.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:01:28.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2021-07-26T07:01:30.234 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:01:30.234 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:01:30.235 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:01:30.235 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s0_head,0#3:00000000::::head#, (61) No data available 2021-07-26T07:01:30.235 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:01:30.235 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:01:30.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:01:30.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:01:30.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:01:30.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:01:30.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:01:30.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:01:30.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:01:30.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:01:30.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:01:30.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:01:30.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:01:30.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:01:30.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:01:30.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:01:30.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:01:30.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:01:30.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:01:30.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:01:30.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:01:30.758 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:01:30.759 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:01:30.759 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:01:30.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:01:30.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:01:30.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:01:30.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:01:30.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:01:30.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:01:30.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:01:30.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:01:30.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:01:30.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:01:30.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:01:30.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:01:30.762 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:01:30.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:01:30.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:01:30.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:01:30.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:01:30.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:01:30.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:01:30.778 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:30.778+0000 7ff486c970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:30.780 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:30.779+0000 7ff486c970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:30.781 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:30.780+0000 7ff486c970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:31.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:01:31.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:01:31.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:01:31.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:01:31.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:01:31.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:31.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:01:31.034 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:01:31.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:31.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:01:31.297 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:31.297+0000 7ff486c970c0 -1 Falling back to public interface 2021-07-26T07:01:31.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:01:32.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:01:32.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:32.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:01:32.300 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:01:32.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:32.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:01:32.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:01:33.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:01:33.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:33.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:01:33.572 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:01:33.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:33.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:01:33.730 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:33.730+0000 7ff486c970c0 -1 osd.1 99 log_to_monitors {default=true} 2021-07-26T07:01:33.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:01:34.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:01:34.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:34.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:01:34.848 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:01:34.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:34.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:01:35.201 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 104 up_thru 104 down_at 100 last_clean_interval [88,99) [v2:127.0.0.1:6810/329868,v1:127.0.0.1:6811/329868] [v2:127.0.0.1:6812/329868,v1:127.0.0.1:6813/329868] exists,up 3ae1a832-ab51-45a4-92a1-c129a1b4b389 2021-07-26T07:01:35.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:01:35.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:01:35.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:01:35.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:01:35.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:01:35.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:01:35.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:01:35.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:01:35.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:01:35.204 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:01:35.204 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:01:35.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:01:35.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:01:35.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:01:35.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:01:35.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:01:35.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:01:35.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:01:35.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:01:35.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:01:35.564 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:01:35.564 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:01:35.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:01:35.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:01:35.565 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:01:35.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=416611827717 2021-07-26T07:01:35.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 416611827717 2021-07-26T07:01:35.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-416611827717' 2021-07-26T07:01:35.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:01:35.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:01:35.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=446676598786 2021-07-26T07:01:35.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 446676598786 2021-07-26T07:01:35.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-416611827717 1-446676598786' 2021-07-26T07:01:35.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:01:35.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:01:35.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=339302416397 2021-07-26T07:01:35.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 339302416397 2021-07-26T07:01:35.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-416611827717 1-446676598786 2-339302416397' 2021-07-26T07:01:35.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:01:35.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-416611827717 2021-07-26T07:01:35.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:01:35.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:01:35.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-416611827717 2021-07-26T07:01:35.920 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:01:35.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=416611827717 2021-07-26T07:01:35.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 416611827717' 2021-07-26T07:01:35.920 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 416611827717 2021-07-26T07:01:35.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:01:36.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 416611827716 -lt 416611827717 2021-07-26T07:01:36.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:01:37.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:01:37.190 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:01:37.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 416611827717 -lt 416611827717 2021-07-26T07:01:37.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:01:37.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-446676598786 2021-07-26T07:01:37.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:01:37.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:01:37.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-446676598786 2021-07-26T07:01:37.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:01:37.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=446676598786 2021-07-26T07:01:37.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 446676598786' 2021-07-26T07:01:37.468 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 446676598786 2021-07-26T07:01:37.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:01:37.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 446676598786 -lt 446676598786 2021-07-26T07:01:37.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:01:37.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-339302416397 2021-07-26T07:01:37.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:01:37.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:01:37.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-339302416397 2021-07-26T07:01:37.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:01:37.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=339302416397 2021-07-26T07:01:37.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 339302416397' 2021-07-26T07:01:37.771 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 339302416397 2021-07-26T07:01:37.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:01:38.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 339302416397 -lt 339302416397 2021-07-26T07:01:38.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:01:38.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:38.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:38.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:01:38.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:38.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:38.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:38.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:38.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:38.398 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:38.398 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:38.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:01:38.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:38.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:38.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:39.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:01:39.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:01:39.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:01:39.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2021-07-26T07:01:39.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: objname=EOBJ7 2021-07-26T07:01:39.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2021-07-26T07:01:39.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:01:39.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T07:01:39.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ7 2021-07-26T07:01:39.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:01:39.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:01:39.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:01:39.424 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:01:39.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:01:39.944 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:01:39.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:01:39.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:01:39.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:01:45.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: expr 7 % 2 2021-07-26T07:01:45.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: local osd=1 2021-07-26T07:01:45.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: case $i in 2021-07-26T07:01:45.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2021-07-26T07:01:45.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3681: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2021-07-26T07:01:45.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2021-07-26T07:01:45.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3685: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2021-07-26T07:01:45.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:01:45.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:01:45.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:01:45.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:01:45.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2021-07-26T07:01:45.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:01:45.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:01:45.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:01:45.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:01:45.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:01:45.583 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:01:45.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:01:45.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:01:45.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:01:45.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:01:45.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:01:45.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2021-07-26T07:01:45.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:01:45.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:01:45.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:01:45.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:01:45.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:01:45.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2021-07-26T07:01:47.518 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:01:47.519 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:01:47.519 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:01:47.519 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:01:47.519 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:01:47.519 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s2_head,2#3:00000000::::head#, (61) No data available 2021-07-26T07:01:48.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:01:48.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:01:48.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:01:48.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:01:48.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:01:48.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:01:48.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:01:48.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:01:48.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:01:48.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:01:48.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:01:48.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:01:48.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:01:48.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:01:48.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:01:48.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:01:48.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:01:48.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:01:48.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:01:48.040 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:01:48.040 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:01:48.041 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:01:48.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:01:48.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:01:48.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:01:48.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:01:48.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:01:48.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:01:48.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:01:48.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:01:48.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:01:48.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:01:48.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:01:48.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:01:48.043 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:01:48.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:01:48.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:01:48.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:01:48.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:01:48.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:01:48.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:01:48.061 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:48.060+0000 7f0a0d2f40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:48.063 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:48.062+0000 7f0a0d2f40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:48.064 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:48.063+0000 7f0a0d2f40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:48.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:01:48.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:01:48.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:01:48.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:01:48.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:01:48.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:48.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:01:48.316 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:01:48.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:48.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:01:48.592 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:48.590+0000 7f0a0d2f40c0 -1 Falling back to public interface 2021-07-26T07:01:48.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:01:49.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:01:49.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:49.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:01:49.613 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:01:49.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:49.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:01:49.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:01:50.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:01:50.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:50.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:01:50.891 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:01:50.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:50.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:01:51.140 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:51.139+0000 7f0a0d2f40c0 -1 osd.0 107 log_to_monitors {default=true} 2021-07-26T07:01:51.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:01:52.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:01:52.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:52.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:01:52.229 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:01:52.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:52.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:01:52.586 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 112 up_thru 112 down_at 108 last_clean_interval [97,107) [v2:127.0.0.1:6802/331303,v1:127.0.0.1:6803/331303] [v2:127.0.0.1:6804/331303,v1:127.0.0.1:6805/331303] exists,up c3ed524e-cfd1-4ff6-b405-a9438f239b1a 2021-07-26T07:01:52.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:01:52.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:01:52.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:01:52.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:01:52.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:01:52.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:01:52.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:01:52.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:01:52.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:01:52.589 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:01:52.589 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:01:52.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:01:52.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:01:52.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:01:52.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:01:52.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:01:52.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:01:52.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:01:52.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:01:52.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:01:52.954 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:01:52.954 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:01:52.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:01:52.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:01:52.955 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:01:53.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=481036337154 2021-07-26T07:01:53.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 481036337154 2021-07-26T07:01:53.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-481036337154' 2021-07-26T07:01:53.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:01:53.074 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:01:53.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=446676598790 2021-07-26T07:01:53.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 446676598790 2021-07-26T07:01:53.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-481036337154 1-446676598790' 2021-07-26T07:01:53.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:01:53.266 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:01:53.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=339302416401 2021-07-26T07:01:53.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 339302416401 2021-07-26T07:01:53.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-481036337154 1-446676598790 2-339302416401' 2021-07-26T07:01:53.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:01:53.385 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-481036337154 2021-07-26T07:01:53.385 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:01:53.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:01:53.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-481036337154 2021-07-26T07:01:53.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:01:53.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=481036337154 2021-07-26T07:01:53.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 481036337154' 2021-07-26T07:01:53.388 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 481036337154 2021-07-26T07:01:53.389 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:01:53.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 481036337154 2021-07-26T07:01:53.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:01:54.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:01:54.652 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:01:54.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 481036337154 -lt 481036337154 2021-07-26T07:01:54.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:01:54.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-446676598790 2021-07-26T07:01:54.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:01:54.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:01:54.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-446676598790 2021-07-26T07:01:54.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:01:54.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=446676598790 2021-07-26T07:01:54.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 446676598790' 2021-07-26T07:01:54.969 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 446676598790 2021-07-26T07:01:54.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:01:55.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 446676598791 -lt 446676598790 2021-07-26T07:01:55.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:01:55.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-339302416401 2021-07-26T07:01:55.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:01:55.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:01:55.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-339302416401 2021-07-26T07:01:55.238 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:01:55.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=339302416401 2021-07-26T07:01:55.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 339302416401' 2021-07-26T07:01:55.239 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 339302416401 2021-07-26T07:01:55.239 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:01:55.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 339302416402 -lt 339302416401 2021-07-26T07:01:55.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:01:55.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:55.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:55.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:01:55.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:55.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:55.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:55.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:55.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:55.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:55.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:56.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:01:56.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:56.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:56.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:56.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:01:56.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:01:56.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:01:56.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2021-07-26T07:01:56.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:01:56.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:01:56.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:01:56.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:01:56.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2021-07-26T07:01:56.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:01:56.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:01:56.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:01:56.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:01:56.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:01:56.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:01:56.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:01:56.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:01:56.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:01:56.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:01:56.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:01:56.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2021-07-26T07:01:56.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:01:56.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:01:56.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:01:56.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:01:56.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:01:56.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2021-07-26T07:01:58.351 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:01:58.352 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:01:58.352 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:01:58.352 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:01:58.352 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:01:58.353 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0s2_head,2#3:00000000::::head#, (61) No data available 2021-07-26T07:01:58.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:01:58.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:01:58.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:01:58.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:01:58.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:01:58.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:01:58.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:01:58.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:01:58.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:01:58.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:01:58.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:01:58.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:01:58.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:01:58.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:01:58.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:01:58.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:01:58.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:01:58.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:01:58.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:01:58.876 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:01:58.876 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:01:58.876 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:01:58.876 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:01:58.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:01:58.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:01:58.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:01:58.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:01:58.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:01:58.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:01:58.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:01:58.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:01:58.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:01:58.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:01:58.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:01:58.880 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:01:58.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:01:58.880 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:01:58.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:01:58.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:01:58.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:01:58.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:01:58.898 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:58.896+0000 7f43f9fe70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:58.899 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:58.898+0000 7f43f9fe70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:58.899 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:58.899+0000 7f43f9fe70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:01:59.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:01:59.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:01:59.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:01:59.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:01:59.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:01:59.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:01:59.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:01:59.156 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:01:59.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:01:59.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:01:59.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:01:59.429 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:01:59.428+0000 7f43f9fe70c0 -1 Falling back to public interface 2021-07-26T07:02:00.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:02:00.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:02:00.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:02:00.418 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:02:00.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:02:00.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:02:00.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:02:01.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:02:01.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:02:01.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:02:01.736 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:02:01.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:02:01.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:02:01.933 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:01.933+0000 7f43f9fe70c0 -1 osd.0 114 log_to_monitors {default=true} 2021-07-26T07:02:02.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:02:03.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:02:03.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:02:03.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:02:03.031 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:02:03.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:02:03.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:02:03.338 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 119 up_thru 119 down_at 115 last_clean_interval [112,114) [v2:127.0.0.1:6802/332651,v1:127.0.0.1:6803/332651] [v2:127.0.0.1:6804/332651,v1:127.0.0.1:6805/332651] exists,up c3ed524e-cfd1-4ff6-b405-a9438f239b1a 2021-07-26T07:02:03.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:02:03.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:02:03.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:02:03.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:02:03.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:02:03.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:02:03.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:02:03.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:02:03.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:02:03.341 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:02:03.341 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:02:03.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:02:03.342 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:02:03.342 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:02:03.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:02:03.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:02:03.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:02:03.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:02:03.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:02:03.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:02:03.742 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:02:03.742 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:02:03.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:02:03.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:02:03.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:02:03.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=511101108226 2021-07-26T07:02:03.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 511101108226 2021-07-26T07:02:03.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-511101108226' 2021-07-26T07:02:03.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:02:03.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:02:04.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=446676598794 2021-07-26T07:02:04.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 446676598794 2021-07-26T07:02:04.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-511101108226 1-446676598794' 2021-07-26T07:02:04.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:02:04.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:02:04.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=339302416405 2021-07-26T07:02:04.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 339302416405 2021-07-26T07:02:04.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-511101108226 1-446676598794 2-339302416405' 2021-07-26T07:02:04.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:02:04.153 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-511101108226 2021-07-26T07:02:04.154 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:02:04.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:02:04.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-511101108226 2021-07-26T07:02:04.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:02:04.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=511101108226 2021-07-26T07:02:04.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 511101108226' 2021-07-26T07:02:04.156 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 511101108226 2021-07-26T07:02:04.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:02:04.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 511101108226 2021-07-26T07:02:04.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:02:05.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:02:05.422 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:02:05.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 511101108226 -lt 511101108226 2021-07-26T07:02:05.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:02:05.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-446676598794 2021-07-26T07:02:05.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:02:05.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:02:05.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-446676598794 2021-07-26T07:02:05.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:02:05.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=446676598794 2021-07-26T07:02:05.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 446676598794' 2021-07-26T07:02:05.703 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 446676598794 2021-07-26T07:02:05.703 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:02:05.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 446676598794 -lt 446676598794 2021-07-26T07:02:05.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:02:05.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-339302416405 2021-07-26T07:02:05.972 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:02:05.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:02:05.973 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-339302416405 2021-07-26T07:02:05.973 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:02:05.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=339302416405 2021-07-26T07:02:05.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 339302416405' 2021-07-26T07:02:05.974 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.2 seq 339302416405 2021-07-26T07:02:05.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:02:06.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 339302416405 -lt 339302416405 2021-07-26T07:02:06.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:02:06.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:06.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:06.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:02:06.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:06.649 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:06.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:06.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:06.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:06.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:06.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:06.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:02:06.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:06.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:06.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:07.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:02:07.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:02:07.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:02:07.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3687: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2021-07-26T07:02:07.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3693: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2021-07-26T07:02:07.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T07:02:07.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=EOBJ0 2021-07-26T07:02:07.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool EOBJ0 2021-07-26T07:02:07.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:02:07.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3693: corrupt_scrub_erasure: local pg=3.0 2021-07-26T07:02:07.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3695: corrupt_scrub_erasure: pg_scrub 3.0 2021-07-26T07:02:07.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: pg_scrub: local pgid=3.0 2021-07-26T07:02:07.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: pg_scrub: get_last_scrub_stamp 3.0 2021-07-26T07:02:07.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:02:07.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:02:07.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:02:07.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:02:07.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: pg_scrub: local last_scrub=2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:07.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: pg_scrub: ceph pg scrub 3.0 2021-07-26T07:02:08.102 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 3.0s0 on osd.1 to scrub 2021-07-26T07:02:08.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: pg_scrub: wait_for_scrub 3.0 2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:08.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=3.0 2021-07-26T07:02:08.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:08.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:02:08.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:02:08.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:02:08.114 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:02:08.114 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:02:08.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:02:08.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:02:08.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:02:08.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:59:29.021168+0000 '>' 2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:08.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:02:09.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:02:09.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:02:09.374 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:02:09.374 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:02:09.375 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:02:09.375 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:02:09.375 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:02:09.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:59:29.021168+0000 '>' 2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:09.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:02:10.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:02:10.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:02:10.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:02:10.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:02:10.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:02:10.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:02:10.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:02:10.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:59:29.021168+0000 '>' 2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:10.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:02:11.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:02:11.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:02:11.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:02:11.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:02:11.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:02:11.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:02:11.920 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:02:12.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:59:29.021168+0000 '>' 2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:12.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:02:13.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:02:13.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:02:13.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:02:13.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:02:13.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:02:13.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:02:13.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:02:13.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:59:29.021168+0000 '>' 2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:13.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:02:14.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:02:14.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:02:14.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:02:14.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:02:14.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:02:14.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:02:14.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:02:14.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:02:11.546520+0000 '>' 2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:14.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:02:14.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3697: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2021-07-26T07:02:14.792 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3699: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2021-07-26T07:02:14.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3699: corrupt_scrub_erasure: test 1 = 1 2021-07-26T07:02:14.794 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3701: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2021-07-26T07:02:14.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3701: corrupt_scrub_erasure: test 3.0 = 3.0 2021-07-26T07:02:14.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3703: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2021-07-26T07:02:14.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3705: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2021-07-26T07:02:14.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3705: corrupt_scrub_erasure: epoch=119 2021-07-26T07:02:14.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3707: corrupt_scrub_erasure: jq 'def walk(f): 2021-07-26T07:02:14.823 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T07:02:14.823 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T07:02:14.823 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T07:02:14.823 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T07:02:14.824 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T07:02:14.824 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T07:02:14.824 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T07:02:14.825 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T07:02:14.826 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T07:02:14.826 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T07:02:14.826 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T07:02:14.827 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2021-07-26T07:02:14.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3707: corrupt_scrub_erasure: jq .inconsistents 2021-07-26T07:02:14.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3707: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T07:02:14.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4327: corrupt_scrub_erasure: jq 'def walk(f): 2021-07-26T07:02:14.848 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T07:02:14.848 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T07:02:14.848 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T07:02:14.849 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T07:02:14.849 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T07:02:14.849 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T07:02:14.849 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T07:02:14.849 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T07:02:14.850 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T07:02:14.850 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T07:02:14.851 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T07:02:14.851 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2021-07-26T07:02:14.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4327: corrupt_scrub_erasure: jq .inconsistents 2021-07-26T07:02:14.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4327: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T07:02:14.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T07:02:14.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T07:02:14.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4329: corrupt_scrub_erasure: test no = yes 2021-07-26T07:02:14.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: test '' = yes 2021-07-26T07:02:14.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4339: corrupt_scrub_erasure: pg_deep_scrub 3.0 2021-07-26T07:02:14.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: pg_deep_scrub: local pgid=3.0 2021-07-26T07:02:14.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:02:14.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:02:14.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:02:14.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:02:14.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:02:15.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: pg_deep_scrub: local last_scrub=2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:15.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1800: pg_deep_scrub: ceph pg deep-scrub 3.0 2021-07-26T07:02:15.388 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2021-07-26T07:02:15.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: pg_deep_scrub: wait_for_scrub 3.0 2021-07-26T06:59:29.021168+0000 last_deep_scrub_stamp 2021-07-26T07:02:15.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=3.0 2021-07-26T07:02:15.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:15.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_deep_scrub_stamp 2021-07-26T07:02:15.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:02:15.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:02:15.400 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:02:15.400 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:02:15.400 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:02:15.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:02:15.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:02:15.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:59:29.021168+0000 '>' 2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:15.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:02:16.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:02:16.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:02:16.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:02:16.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:02:16.662 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:02:16.662 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:02:16.662 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:02:16.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:59:29.021168+0000 '>' 2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:16.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:02:17.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:02:17.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:02:17.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:02:17.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:02:17.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:02:17.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:02:17.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:02:18.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:59:29.021168+0000 '>' 2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:18.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:02:19.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:02:19.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:02:19.207 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:02:19.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:02:19.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:02:19.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:02:19.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:02:19.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T06:59:29.021168+0000 '>' 2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:19.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:02:20.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:02:20.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:02:20.485 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:02:20.485 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:02:20.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:02:20.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:02:20.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:02:20.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:02:17.637106+0000 '>' 2021-07-26T06:59:29.021168+0000 2021-07-26T07:02:20.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:02:20.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4341: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2021-07-26T07:02:20.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4343: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2021-07-26T07:02:20.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4343: corrupt_scrub_erasure: test 1 = 1 2021-07-26T07:02:20.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4345: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2021-07-26T07:02:20.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4345: corrupt_scrub_erasure: test 3.0 = 3.0 2021-07-26T07:02:20.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4347: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2021-07-26T07:02:20.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4349: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2021-07-26T07:02:20.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4349: corrupt_scrub_erasure: epoch=119 2021-07-26T07:02:20.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4351: corrupt_scrub_erasure: '[' true = true ']' 2021-07-26T07:02:20.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4353: corrupt_scrub_erasure: jq 'def walk(f): 2021-07-26T07:02:20.814 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T07:02:20.814 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T07:02:20.815 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T07:02:20.815 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T07:02:20.815 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T07:02:20.815 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T07:02:20.815 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T07:02:20.816 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T07:02:20.816 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T07:02:20.816 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T07:02:20.816 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T07:02:20.817 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2021-07-26T07:02:20.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4353: corrupt_scrub_erasure: jq .inconsistents 2021-07-26T07:02:20.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4353: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T07:02:20.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5741: corrupt_scrub_erasure: jq 'def walk(f): 2021-07-26T07:02:20.838 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T07:02:20.838 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T07:02:20.839 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T07:02:20.839 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T07:02:20.839 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T07:02:20.840 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T07:02:20.840 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T07:02:20.840 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T07:02:20.840 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T07:02:20.840 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T07:02:20.841 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T07:02:20.841 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2021-07-26T07:02:20.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5741: corrupt_scrub_erasure: jq .inconsistents 2021-07-26T07:02:20.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5741: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T07:02:20.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T07:02:20.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T07:02:20.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5743: corrupt_scrub_erasure: test no = yes 2021-07-26T07:02:20.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5754: corrupt_scrub_erasure: test '' = yes 2021-07-26T07:02:20.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2021-07-26T07:02:21.319 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' removed 2021-07-26T07:02:21.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T07:02:21.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:02:21.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:02:21.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:02:21.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:02:21.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:02:21.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:02:21.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:02:21.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:02:21.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:02:21.438 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:02:21.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:02:21.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:02:21.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:02:21.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:02:21.440 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:02:21.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:02:21.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:02:21.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:02:21.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:02:21.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:02:21.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:02:21.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:02:21.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:02:21.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:02:21.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:02:21.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:02:21.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:02:21.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:02:21.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T07:02:21.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T07:02:21.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T07:02:21.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T07:02:21.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:02:21.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:02:21.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:02:21.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:02:21.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:02:21.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:02:21.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:02:21.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:02:21.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:02:21.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:02:21.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:02:21.471 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:02:21.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:02:21.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:02:21.471 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:02:21.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:02:21.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:02:21.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:02:21.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:02:21.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:02:21.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:02:21.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:02:21.475 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:02:21.475 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:02:21.476 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:02:21.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:02:21.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:02:21.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:02:21.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T07:02:21.479 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:02:21.479 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:02:21.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:02:21.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T07:02:21.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:02:21.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:02:21.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:02:21.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T07:02:21.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2021-07-26T07:02:21.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2021-07-26T07:02:21.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1005: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2021-07-26T07:02:21.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: local total_objs=19 2021-07-26T07:02:21.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2021-07-26T07:02:21.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T07:02:21.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:02:21.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:02:21.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:02:21.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T07:02:21.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2021-07-26T07:02:21.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:02:21.841 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:02:21.841 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:02:21.841 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:02:21.842 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:02:21.842 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:02:21.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:02:21.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2021-07-26T07:02:22.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:02:22.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:02:22.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:02:22.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:02:22.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:02:22.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:02:22.095 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:02:22.095 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:02:22.096 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:02:22.096 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:02:22.096 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:02:22.096 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:02:22.097 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:02:22.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:02:22.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T07:02:22.193 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:02:22.193 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:02:22.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:02:22.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:02:22.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:02:22.195 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:02:22.195 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:02:22.195 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:02:22.196 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:02:22.197 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:02:22.197 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:02:22.198 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:02:22.198 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:02:22.198 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T07:02:22.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1009: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2021-07-26T07:02:22.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T07:02:22.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:02:22.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:02:22.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:02:22.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T07:02:22.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:02:23.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:02:23.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:02:23.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:02:23.121 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:02:23.121 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:02:23.121 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:02:23.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:02:23.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:02:23.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:02:23.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2021-07-26T07:02:23.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:02:23.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:02:23.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:02:23.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:02:23.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:02:23.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:02:23.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:02:23.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:02:23.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:02:23.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:02:23.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:02:23.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:02:23.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:02:23.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:02:23.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:02:23.151 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:02:23.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:02:23.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:02:23.152 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:02:23.152 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:02:23.153 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:02:23.153 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:02:23.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:02:23.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:02:23.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:02:23.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:02:23.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:02:23.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:02:23.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:02:23.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:02:23.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:02:23.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:02:23.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:02:23.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:02:23.156 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:02:23.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:02:23.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5e04f48f-0d4c-440e-9f99-bf7e8b923c15' 2021-07-26T07:02:23.157 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:02:23.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:02:23.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/Xf5g7xenCRAAQ1OZaz21oe9weQT3ka3AbQ== 2021-07-26T07:02:23.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/Xf5g7xenCRAAQ1OZaz21oe9weQT3ka3AbQ=="}' 2021-07-26T07:02:23.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 -i td/osd-scrub-repair/0/new.json 2021-07-26T07:02:23.417 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:02:23.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T07:02:23.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD/Xf5g7xenCRAAQ1OZaz21oe9weQT3ka3AbQ== --osd-uuid 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:02:23.452 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:23.452+0000 7f9bff5640c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:02:23.454 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:23.454+0000 7f9bff5640c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:02:23.456 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:23.455+0000 7f9bff5640c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:02:23.456 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:23.456+0000 7f9bff5640c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:02:23.456 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:23.456+0000 7f9bff5640c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:02:23.457 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:23.456+0000 7f9bff5640c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:02:23.534 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:23.533+0000 7f9bff5640c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T07:02:29.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T07:02:29.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:02:29.259 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:02:29.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:02:29.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:02:29.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:02:29.599 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:02:29.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:02:29.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:02:29.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:02:29.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:02:29.620 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:29.620+0000 7fbbafd2f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:02:29.622 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:29.622+0000 7fbbafd2f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:02:29.623 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:29.623+0000 7fbbafd2f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:02:29.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:02:29.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:02:29.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:02:29.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:02:29.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:02:29.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:02:29.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:02:29.911 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:02:29.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:02:29.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:02:30.137 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:30.137+0000 7fbbafd2f0c0 -1 Falling back to public interface 2021-07-26T07:02:30.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:02:31.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:02:31.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:02:31.210 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:02:31.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:02:31.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:02:31.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:02:31.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:02:32.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:02:32.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:02:32.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:02:32.487 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:02:32.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:02:32.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:02:32.532 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:32.532+0000 7fbbafd2f0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:02:32.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:02:33.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:02:33.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:02:33.799 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:02:33.800 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:02:33.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:02:33.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:02:34.076 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/335272,v1:127.0.0.1:6803/335272] [v2:127.0.0.1:6804/335272,v1:127.0.0.1:6805/335272] exists,up 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:02:34.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:02:34.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:02:34.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:02:34.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2021-07-26T07:02:34.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:02:34.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:02:34.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:02:34.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:02:34.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:02:34.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:02:34.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:02:34.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:02:34.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:02:34.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:02:34.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:02:34.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:02:34.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:02:34.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:02:34.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:02:34.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:02:34.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:02:34.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:02:34.086 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:02:34.086 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:02:34.087 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:02:34.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:02:34.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:02:34.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:02:34.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:02:34.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:02:34.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:02:34.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:02:34.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:02:34.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:02:34.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:02:34.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:02:34.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:02:34.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:02:34.090 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:02:34.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:02:34.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f9d715e4-5f23-432c-b848-af796cb0a07b' 2021-07-26T07:02:34.091 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:02:34.091 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:02:34.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAKXv5gRjOzBRAAxfYq89gqA7OitnitFxYG0w== 2021-07-26T07:02:34.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAKXv5gRjOzBRAAxfYq89gqA7OitnitFxYG0w=="}' 2021-07-26T07:02:34.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f9d715e4-5f23-432c-b848-af796cb0a07b -i td/osd-scrub-repair/1/new.json 2021-07-26T07:02:34.468 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:02:34.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T07:02:34.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAKXv5gRjOzBRAAxfYq89gqA7OitnitFxYG0w== --osd-uuid f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:02:34.504 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:34.503+0000 7f0be79bf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:02:34.506 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:34.504+0000 7f0be79bf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:02:34.506 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:34.505+0000 7f0be79bf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:02:34.507 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:34.506+0000 7f0be79bf0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:02:34.508 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:34.506+0000 7f0be79bf0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:02:34.508 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:34.506+0000 7f0be79bf0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:02:34.578 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:34.577+0000 7f0be79bf0c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T07:02:40.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T07:02:40.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:02:40.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:02:40.059 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T07:02:40.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:02:40.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:02:40.398 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T07:02:40.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:02:40.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:02:40.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:02:40.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:02:40.419 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:40.418+0000 7f19db19e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:02:40.421 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:40.420+0000 7f19db19e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:02:40.422 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:40.421+0000 7f19db19e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:02:40.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:02:40.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:02:40.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:02:40.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:02:40.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:02:40.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:02:40.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:02:40.711 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:02:40.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:02:40.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:02:40.940 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:40.938+0000 7f19db19e0c0 -1 Falling back to public interface 2021-07-26T07:02:40.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:02:41.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:02:41.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:02:41.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:02:41.983 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:02:41.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:02:41.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:02:42.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:02:43.242 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:02:43.240+0000 7f19db19e0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:02:43.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:02:43.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:02:43.259 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:02:43.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:02:43.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:02:43.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:02:43.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:02:44.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:02:44.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:02:44.660 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:02:44.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:02:44.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:02:44.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:02:44.934 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/336671,v1:127.0.0.1:6811/336671] [v2:127.0.0.1:6812/336671,v1:127.0.0.1:6813/336671] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:02:44.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:02:44.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:02:44.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:02:44.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1012: TEST_corrupt_scrub_replicated: create_rbd_pool 2021-07-26T07:02:44.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T07:02:45.226 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T07:02:45.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T07:02:45.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T07:02:45.656 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T07:02:45.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:02:46.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T07:02:48.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1013: TEST_corrupt_scrub_replicated: wait_for_clean 2021-07-26T07:02:48.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:02:48.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:02:48.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:02:48.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:02:48.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:02:48.205 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:02:48.205 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:02:48.206 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:02:48.207 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:02:48.207 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:02:48.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:02:48.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:02:48.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:02:48.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:02:48.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:02:48.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:02:48.576 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:02:48.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:02:48.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:02:48.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:02:48.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836485 2021-07-26T07:02:48.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836485 2021-07-26T07:02:48.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836485' 2021-07-26T07:02:48.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:02:48.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:02:48.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672962 2021-07-26T07:02:48.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672962 2021-07-26T07:02:48.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836485 1-42949672962' 2021-07-26T07:02:48.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:02:48.816 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836485 2021-07-26T07:02:48.816 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:02:48.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:02:48.817 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836485 2021-07-26T07:02:48.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:02:48.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836485 2021-07-26T07:02:48.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2021-07-26T07:02:48.819 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836485 2021-07-26T07:02:48.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:02:49.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836484 -lt 21474836485 2021-07-26T07:02:49.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:02:50.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:02:50.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:02:50.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836485 2021-07-26T07:02:50.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:02:50.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672962 2021-07-26T07:02:50.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:02:50.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:02:50.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672962 2021-07-26T07:02:50.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:02:50.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672962 2021-07-26T07:02:50.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2021-07-26T07:02:50.365 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672962 2021-07-26T07:02:50.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:02:50.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672963 -lt 42949672962 2021-07-26T07:02:50.672 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:02:50.672 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:50.672 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:51.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T07:02:51.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:51.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:51.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:51.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:51.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:51.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:51.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:51.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T07:02:51.342 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:51.343 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:51.343 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:51.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T07:02:51.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:02:51.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:02:51.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: create_pool foo 1 2021-07-26T07:02:51.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2021-07-26T07:02:52.184 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'foo' created 2021-07-26T07:02:52.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:02:53.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2021-07-26T07:02:53.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2021-07-26T07:02:53.609 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'csr_pool' created 2021-07-26T07:02:53.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:02:54.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: wait_for_clean 2021-07-26T07:02:54.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:02:54.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:02:54.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:02:54.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:02:54.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:02:54.623 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:02:54.624 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:02:54.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:02:54.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:02:54.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:02:54.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:02:54.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:02:54.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:02:54.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:02:54.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:02:55.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:02:55.000 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:02:55.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:02:55.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:02:55.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:02:55.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836488 2021-07-26T07:02:55.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836488 2021-07-26T07:02:55.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488' 2021-07-26T07:02:55.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:02:55.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:02:55.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672965 2021-07-26T07:02:55.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672965 2021-07-26T07:02:55.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2021-07-26T07:02:55.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:02:55.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836488 2021-07-26T07:02:55.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:02:55.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:02:55.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836488 2021-07-26T07:02:55.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:02:55.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836488 2021-07-26T07:02:55.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-07-26T07:02:55.278 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836488 2021-07-26T07:02:55.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:02:55.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836488 2021-07-26T07:02:55.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:02:56.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:02:56.543 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:02:56.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836488 -lt 21474836488 2021-07-26T07:02:56.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:02:56.816 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672965 2021-07-26T07:02:56.817 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:02:56.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:02:56.817 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672965 2021-07-26T07:02:56.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:02:56.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672965 2021-07-26T07:02:56.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-07-26T07:02:56.819 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672965 2021-07-26T07:02:56.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:02:57.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672965 -lt 42949672965 2021-07-26T07:02:57.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:02:57.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:57.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:57.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:02:57.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:57.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:57.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:57.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:57.490 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:57.490 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:57.490 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:57.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:02:57.910 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:57.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:57.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:58.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:02:58.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:02:58.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:02:58.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: seq 1 19 2021-07-26T07:02:58.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:02:58.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ1 2021-07-26T07:02:58.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2021-07-26T07:02:58.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:02:58.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:02:58.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ1 2021-07-26T07:02:58.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:02:58.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:02:58.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:02:58.552 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:02:58.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:02:59.097 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:02:59.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:02:59.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:02:59.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:02:59.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2021-07-26T07:02:59.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2021-07-26T07:02:59.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:02:59.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ2 2021-07-26T07:02:59.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2021-07-26T07:02:59.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:02:59.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:02:59.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ2 2021-07-26T07:02:59.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:02:59.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:02:59.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:00.036 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:00.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:00.552 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:00.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:00.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:00.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:00.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2021-07-26T07:03:00.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2021-07-26T07:03:00.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:00.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ3 2021-07-26T07:03:00.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2021-07-26T07:03:00.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:00.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:00.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ3 2021-07-26T07:03:00.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:00.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:00.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:01.372 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:01.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:01.932 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:01.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:01.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:01.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:02.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2021-07-26T07:03:02.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2021-07-26T07:03:02.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:02.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ4 2021-07-26T07:03:02.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2021-07-26T07:03:02.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:02.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:02.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ4 2021-07-26T07:03:02.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:02.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:02.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:02.853 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:02.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:03.372 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:03.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:03.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:03.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:03.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2021-07-26T07:03:03.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2021-07-26T07:03:03.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:03.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ5 2021-07-26T07:03:03.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2021-07-26T07:03:03.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:03.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:03.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ5 2021-07-26T07:03:03.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:03.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:03.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:04.217 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:04.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:04.786 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:04.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:04.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:04.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:04.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2021-07-26T07:03:05.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2021-07-26T07:03:05.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:05.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ6 2021-07-26T07:03:05.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2021-07-26T07:03:05.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:05.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:05.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ6 2021-07-26T07:03:05.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:05.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:05.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:05.580 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:05.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:06.032 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:06.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:06.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:06.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:06.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2021-07-26T07:03:06.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2021-07-26T07:03:06.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:06.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ7 2021-07-26T07:03:06.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2021-07-26T07:03:06.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:06.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:06.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ7 2021-07-26T07:03:06.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:06.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:06.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:06.902 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:06.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:07.446 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:07.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:07.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:07.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:07.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2021-07-26T07:03:07.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2021-07-26T07:03:07.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:07.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ8 2021-07-26T07:03:07.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2021-07-26T07:03:07.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:07.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:07.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ8 2021-07-26T07:03:07.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:07.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:07.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:08.375 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:08.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:08.911 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:08.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:08.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:08.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:09.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2021-07-26T07:03:09.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2021-07-26T07:03:09.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:09.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ9 2021-07-26T07:03:09.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2021-07-26T07:03:09.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:09.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:09.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ9 2021-07-26T07:03:09.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:09.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:09.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:09.805 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:09.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:10.341 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:10.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:10.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:10.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:10.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2021-07-26T07:03:10.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2021-07-26T07:03:10.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:10.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ10 2021-07-26T07:03:10.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2021-07-26T07:03:10.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:10.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:10.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ10 2021-07-26T07:03:10.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:10.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:10.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:11.211 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:11.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:11.747 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:11.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:11.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:11.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:11.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2021-07-26T07:03:12.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2021-07-26T07:03:12.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:12.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ11 2021-07-26T07:03:12.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2021-07-26T07:03:12.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:12.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:12.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ11 2021-07-26T07:03:12.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:12.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:12.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:12.617 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:12.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:13.136 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:13.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:13.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:13.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:13.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2021-07-26T07:03:13.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2021-07-26T07:03:13.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:13.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ12 2021-07-26T07:03:13.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2021-07-26T07:03:13.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:13.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:13.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ12 2021-07-26T07:03:13.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:13.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:13.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:14.059 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:14.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:14.625 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:14.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:14.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:14.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:14.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2021-07-26T07:03:14.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2021-07-26T07:03:15.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:15.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ13 2021-07-26T07:03:15.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2021-07-26T07:03:15.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:15.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:15.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ13 2021-07-26T07:03:15.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:15.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:15.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:15.445 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:15.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:15.963 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:15.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:15.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:15.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:16.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2021-07-26T07:03:16.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2021-07-26T07:03:16.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:16.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ14 2021-07-26T07:03:16.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2021-07-26T07:03:16.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:16.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:16.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ14 2021-07-26T07:03:16.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:16.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:16.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:16.861 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:16.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:17.388 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:17.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:17.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:17.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:17.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2021-07-26T07:03:17.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2021-07-26T07:03:17.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:17.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ15 2021-07-26T07:03:17.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2021-07-26T07:03:17.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:17.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:17.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ15 2021-07-26T07:03:17.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:17.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:17.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:18.250 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:18.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:18.770 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:18.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:18.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:18.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:18.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2021-07-26T07:03:19.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2021-07-26T07:03:19.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:19.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ16 2021-07-26T07:03:19.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2021-07-26T07:03:19.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:19.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:19.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ16 2021-07-26T07:03:19.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:19.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:19.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:19.565 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:19.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:20.101 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:20.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:20.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:20.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:20.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2021-07-26T07:03:20.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2021-07-26T07:03:20.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:20.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ17 2021-07-26T07:03:20.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2021-07-26T07:03:20.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:20.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:20.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ17 2021-07-26T07:03:20.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:20.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:20.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:20.904 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:20.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:21.423 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:21.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:21.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:21.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:21.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2021-07-26T07:03:21.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2021-07-26T07:03:21.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:21.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ18 2021-07-26T07:03:21.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2021-07-26T07:03:21.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:21.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:21.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ18 2021-07-26T07:03:21.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:21.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:21.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:22.260 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:22.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:22.788 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:22.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:22.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:22.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:23.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2021-07-26T07:03:23.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2021-07-26T07:03:23.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:23.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: objname=ROBJ19 2021-07-26T07:03:23.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2021-07-26T07:03:23.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:03:23.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:03:23.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ19 2021-07-26T07:03:23.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:03:23.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:03:23.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:03:23.608 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:03:23.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:03:24.094 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:03:24.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:03:24.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:03:24.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:03:24.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2021-07-26T07:03:24.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2021-07-26T07:03:24.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2021-07-26T07:03:24.815 INFO:tasks.workunit.client.0.smithi071.stderr:1025+0 records in 2021-07-26T07:03:24.816 INFO:tasks.workunit.client.0.smithi071.stderr:1025+0 records out 2021-07-26T07:03:24.816 INFO:tasks.workunit.client.0.smithi071.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00183121 s, 573 MB/s 2021-07-26T07:03:24.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2021-07-26T07:03:24.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2021-07-26T07:03:24.957 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2021-07-26T07:03:24.957 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=csr_pool 2021-07-26T07:03:24.957 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=ROBJ0 2021-07-26T07:03:24.957 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map csr_pool ROBJ0 2021-07-26T07:03:24.958 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:03:25.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: TEST_corrupt_scrub_replicated: local pg=3.0 2021-07-26T07:03:25.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2021-07-26T07:03:25.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=csr_pool 2021-07-26T07:03:25.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=ROBJ0 2021-07-26T07:03:25.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map csr_pool ROBJ0 2021-07-26T07:03:25.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:03:25.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local primary=1 2021-07-26T07:03:25.494 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1036: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2021-07-26T07:03:25.494 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T07:03:25.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T07:03:25.495 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:03:25.495 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:03:25.495 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:03:25.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.0.asok 2021-07-26T07:03:25.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1036: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2021-07-26T07:03:25.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1036: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.128899/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2021-07-26T07:03:25.587 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:03:25.587 INFO:tasks.workunit.client.0.smithi071.stdout: "success": "osd_deep_scrub_update_digest_min_age = '0' (not observed, change may require restart) " 2021-07-26T07:03:25.588 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:03:25.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2021-07-26T07:03:25.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T07:03:25.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T07:03:25.598 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:03:25.598 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:03:25.598 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:03:25.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.1.asok 2021-07-26T07:03:25.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2021-07-26T07:03:25.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.128899/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2021-07-26T07:03:25.691 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:03:25.692 INFO:tasks.workunit.client.0.smithi071.stdout: "success": "osd_deep_scrub_update_digest_min_age = '0' (not observed, change may require restart) " 2021-07-26T07:03:25.692 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:03:25.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2021-07-26T07:03:25.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: pg_deep_scrub: local pgid=3.0 2021-07-26T07:03:25.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:03:25.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:03:25.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:03:25.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:03:25.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:03:25.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: pg_deep_scrub: local last_scrub=2021-07-26T07:02:53.517779+0000 2021-07-26T07:03:25.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1800: pg_deep_scrub: ceph pg deep-scrub 3.0 2021-07-26T07:03:26.215 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2021-07-26T07:03:26.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: pg_deep_scrub: wait_for_scrub 3.0 2021-07-26T07:02:53.517779+0000 last_deep_scrub_stamp 2021-07-26T07:03:26.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=3.0 2021-07-26T07:03:26.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:02:53.517779+0000 2021-07-26T07:03:26.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_deep_scrub_stamp 2021-07-26T07:03:26.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:03:26.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:03:26.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:03:26.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:03:26.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:03:26.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:03:26.228 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:03:26.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:02:53.517779+0000 '>' 2021-07-26T07:02:53.517779+0000 2021-07-26T07:03:26.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:03:27.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:03:27.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:03:27.488 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:03:27.488 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:03:27.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:03:27.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:03:27.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:03:27.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:02:53.517779+0000 '>' 2021-07-26T07:02:53.517779+0000 2021-07-26T07:03:27.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:03:28.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:03:28.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:03:28.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:03:28.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:03:28.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:03:28.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:03:28.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:03:29.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:02:53.517779+0000 '>' 2021-07-26T07:02:53.517779+0000 2021-07-26T07:03:29.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:03:30.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:03:30.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:03:30.053 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:03:30.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:03:30.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:03:30.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:03:30.055 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:03:30.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:02:53.517779+0000 '>' 2021-07-26T07:02:53.517779+0000 2021-07-26T07:03:30.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:03:31.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:03:31.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:03:31.324 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:03:31.324 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:03:31.324 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:03:31.325 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:03:31.325 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:03:31.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:02:53.517779+0000 '>' 2021-07-26T07:02:53.517779+0000 2021-07-26T07:03:31.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:03:32.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:03:32.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:03:32.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:03:32.630 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:03:32.630 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:03:32.630 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:03:32.630 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:03:32.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:02:53.517779+0000 '>' 2021-07-26T07:02:53.517779+0000 2021-07-26T07:03:32.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:03:33.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:03:33.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:03:33.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:03:33.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:03:33.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:03:33.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:03:33.904 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:03:34.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:02:53.517779+0000 '>' 2021-07-26T07:02:53.517779+0000 2021-07-26T07:03:34.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:03:35.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:03:35.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:03:35.178 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:03:35.178 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:03:35.178 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:03:35.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:03:35.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:03:35.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:02:53.517779+0000 '>' 2021-07-26T07:02:53.517779+0000 2021-07-26T07:03:35.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:03:36.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:03:36.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:03:36.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:03:36.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:03:36.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:03:36.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:03:36.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:03:36.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:03:30.569566+0000 '>' 2021-07-26T07:02:53.517779+0000 2021-07-26T07:03:36.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:03:36.748 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: seq 1 19 2021-07-26T07:03:36.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:36.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ1 2021-07-26T07:03:36.749 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 1 % 2 2021-07-26T07:03:36.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=1 2021-07-26T07:03:36.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:03:36.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1051: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2021-07-26T07:03:36.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2021-07-26T07:03:36.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1053: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:03:36.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:03:36.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:03:36.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:03:36.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:03:36.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:03:36.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:03:36.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:03:36.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:03:36.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:03:36.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:03:36.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:03:36.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:03:36.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:03:36.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:03:36.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:03:36.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:03:36.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:03:36.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:03:36.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:03:36.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:03:36.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:03:36.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:03:36.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:03:38.462 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:03:38.462 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:03:38.462 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:03:38.462 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:03:38.463 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:03:38.463 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2021-07-26T07:03:38.463 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:03:39.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:03:39.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:03:39.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:03:39.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:03:39.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:03:39.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:03:39.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:03:39.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:03:39.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:03:39.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:03:39.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:03:39.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:03:39.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:03:39.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:03:39.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:03:39.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:03:39.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:03:39.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:03:39.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:03:39.040 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:03:39.040 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:03:39.041 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:03:39.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:03:39.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:03:39.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:03:39.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:03:39.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:03:39.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:03:39.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:03:39.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:03:39.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:03:39.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:03:39.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:03:39.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:03:39.044 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:03:39.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:03:39.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:03:39.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:03:39.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:03:39.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:03:39.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:03:39.055 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:03:39.054+0000 7f2afbf730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:03:39.057 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:03:39.056+0000 7f2afbf730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:03:39.059 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:03:39.057+0000 7f2afbf730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:03:39.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:03:39.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:03:39.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:03:39.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:03:39.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:03:39.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:03:39.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:03:39.313 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:03:39.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:03:39.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:03:39.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:03:39.575 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:03:39.574+0000 7f2afbf730c0 -1 Falling back to public interface 2021-07-26T07:03:40.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:03:40.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:03:40.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:03:40.575 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:03:40.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:03:40.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:03:40.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:03:41.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:03:41.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:03:41.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:03:41.854 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:03:41.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:03:41.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:03:41.954 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:03:41.953+0000 7f2afbf730c0 -1 osd.1 63 log_to_monitors {default=true} 2021-07-26T07:03:42.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:03:43.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:03:43.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:03:43.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:03:43.134 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:03:43.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:03:43.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:03:43.427 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 67 up_thru 67 down_at 64 last_clean_interval [10,63) [v2:127.0.0.1:6810/340801,v1:127.0.0.1:6811/340801] [v2:127.0.0.1:6812/340801,v1:127.0.0.1:6813/340801] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:03:43.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:03:43.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:03:43.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:03:43.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:03:43.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:03:43.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:03:43.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:03:43.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:03:43.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:03:43.429 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:03:43.430 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:03:43.430 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:03:43.430 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:03:43.431 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:03:43.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:03:43.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:03:43.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:03:43.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:03:43.526 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:03:43.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:03:43.788 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:03:43.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:03:43.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:03:43.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:03:43.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836499 2021-07-26T07:03:43.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836499 2021-07-26T07:03:43.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836499' 2021-07-26T07:03:43.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:03:43.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:03:44.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=287762808834 2021-07-26T07:03:44.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 287762808834 2021-07-26T07:03:44.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836499 1-287762808834' 2021-07-26T07:03:44.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:03:44.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836499 2021-07-26T07:03:44.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:03:44.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:03:44.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836499 2021-07-26T07:03:44.100 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:03:44.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836499 2021-07-26T07:03:44.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2021-07-26T07:03:44.101 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836499 2021-07-26T07:03:44.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:03:44.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836497 -lt 21474836499 2021-07-26T07:03:44.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:03:45.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:03:45.367 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:03:45.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836497 -lt 21474836499 2021-07-26T07:03:45.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:03:46.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:03:46.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:03:46.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836499 -lt 21474836499 2021-07-26T07:03:46.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:03:46.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-287762808834 2021-07-26T07:03:46.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:03:46.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:03:46.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-287762808834 2021-07-26T07:03:46.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:03:46.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=287762808834 2021-07-26T07:03:46.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2021-07-26T07:03:46.924 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 287762808834 2021-07-26T07:03:46.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:03:47.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 287762808834 -lt 287762808834 2021-07-26T07:03:47.223 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:03:47.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:47.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:47.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:03:47.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:47.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:47.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:47.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:47.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:47.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:47.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:47.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:03:47.894 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:47.894 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:47.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:48.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:03:48.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:03:48.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:03:48.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:48.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ2 2021-07-26T07:03:48.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 2 % 2 2021-07-26T07:03:48.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=0 2021-07-26T07:03:48.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:03:48.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1058: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2021-07-26T07:03:48.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1059: TEST_corrupt_scrub_replicated: echo UVWXYZ 2021-07-26T07:03:48.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1060: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:03:48.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:03:48.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:03:48.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:03:48.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:03:48.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:03:48.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:03:48.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:03:48.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:03:48.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:03:48.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:03:48.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:03:48.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:03:48.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:03:48.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:03:48.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:03:48.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:03:48.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:03:48.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:03:48.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:03:48.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:03:48.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:03:48.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:03:48.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:03:49.877 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:03:49.877 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:03:49.877 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:03:49.878 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:03:49.878 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:03:49.878 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:03:50.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:03:50.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:03:50.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:03:50.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:03:50.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:03:50.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:03:50.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:03:50.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:03:50.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:03:50.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:03:50.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:03:50.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:03:50.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:03:50.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:03:50.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:03:50.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:03:50.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:03:50.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:03:50.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:03:50.468 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:03:50.469 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:03:50.469 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:03:50.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:03:50.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:03:50.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:03:50.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:03:50.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:03:50.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:03:50.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:03:50.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:03:50.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:03:50.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:03:50.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:03:50.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:03:50.472 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:03:50.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:03:50.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:03:50.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:03:50.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:03:50.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:03:50.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:03:50.488 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:03:50.486+0000 7fc52065e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:03:50.490 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:03:50.488+0000 7fc52065e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:03:50.491 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:03:50.489+0000 7fc52065e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:03:50.743 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:03:50.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:03:50.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:03:50.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:03:50.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:03:50.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:03:50.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:03:50.745 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:03:50.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:03:50.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:03:51.005 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:03:51.003+0000 7fc52065e0c0 -1 Falling back to public interface 2021-07-26T07:03:51.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:03:52.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:03:52.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:03:52.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:03:52.048 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:03:52.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:03:52.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:03:52.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:03:53.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:03:53.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:03:53.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:03:53.326 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:03:53.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:03:53.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:03:53.407 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:03:53.406+0000 7fc52065e0c0 -1 osd.0 68 log_to_monitors {default=true} 2021-07-26T07:03:53.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:03:54.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:03:54.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:03:54.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:03:54.673 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:03:54.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:03:54.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:03:54.941 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [5,68) [v2:127.0.0.1:6802/342122,v1:127.0.0.1:6803/342122] [v2:127.0.0.1:6804/342122,v1:127.0.0.1:6805/342122] exists,up 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:03:54.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:03:54.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:03:54.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:03:54.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:03:54.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:03:54.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:03:54.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:03:54.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:03:54.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:03:54.943 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:03:54.943 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:03:54.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:03:54.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:03:54.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:03:55.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:03:55.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:03:55.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:03:55.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:03:55.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:03:55.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:03:55.302 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:03:55.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:03:55.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:03:55.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:03:55.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=309237645314 2021-07-26T07:03:55.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 309237645314 2021-07-26T07:03:55.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-309237645314' 2021-07-26T07:03:55.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:03:55.421 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:03:55.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=287762808837 2021-07-26T07:03:55.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 287762808837 2021-07-26T07:03:55.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-309237645314 1-287762808837' 2021-07-26T07:03:55.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:03:55.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-309237645314 2021-07-26T07:03:55.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:03:55.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:03:55.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-309237645314 2021-07-26T07:03:55.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:03:55.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=309237645314 2021-07-26T07:03:55.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 309237645314' 2021-07-26T07:03:55.542 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 309237645314 2021-07-26T07:03:55.543 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:03:55.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 309237645314 2021-07-26T07:03:55.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:03:56.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:03:56.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:03:57.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 309237645314 -lt 309237645314 2021-07-26T07:03:57.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:03:57.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-287762808837 2021-07-26T07:03:57.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:03:57.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:03:57.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-287762808837 2021-07-26T07:03:57.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:03:57.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=287762808837 2021-07-26T07:03:57.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 287762808837' 2021-07-26T07:03:57.087 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 287762808837 2021-07-26T07:03:57.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:03:57.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 287762808837 -lt 287762808837 2021-07-26T07:03:57.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:03:57.396 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:57.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:57.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:03:57.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:57.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:57.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:57.759 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:57.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:57.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:57.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:58.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:03:58.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:58.023 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:58.023 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:58.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:03:58.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:03:58.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:03:58.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:03:58.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ3 2021-07-26T07:03:58.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 3 % 2 2021-07-26T07:03:58.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=1 2021-07-26T07:03:58.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:03:58.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2021-07-26T07:03:58.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:03:58.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:03:58.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:03:58.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:03:58.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2021-07-26T07:03:58.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:03:58.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:03:58.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:03:58.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:03:58.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:03:58.356 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:03:58.356 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:03:58.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:03:58.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:03:58.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:03:58.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:03:58.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2021-07-26T07:03:58.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:03:58.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:03:58.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:03:58.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:03:58.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:03:58.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2021-07-26T07:03:59.672 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:03:59.672 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:03:59.673 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:03:59.673 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:03:59.673 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:03:59.673 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:03:59.674 INFO:tasks.workunit.client.0.smithi071.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2021-07-26T07:04:00.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:04:00.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:04:00.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:04:00.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:04:00.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:04:00.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:04:00.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:04:00.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:04:00.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:04:00.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:04:00.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:04:00.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:04:00.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:04:00.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:04:00.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:04:00.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:04:00.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:04:00.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:04:00.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:04:00.201 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:04:00.201 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:00.202 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:04:00.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:04:00.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:04:00.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:04:00.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:04:00.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:04:00.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:04:00.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:04:00.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:00.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:04:00.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:04:00.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:04:00.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:04:00.205 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:04:00.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:00.206 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:04:00.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:04:00.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:04:00.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:04:00.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:04:00.218 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:00.218+0000 7fa5ff0460c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:00.237 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:00.237+0000 7fa5ff0460c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:00.238 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:00.238+0000 7fa5ff0460c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:00.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:04:00.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:04:00.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:04:00.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:04:00.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:04:00.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:00.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:04:00.485 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:04:00.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:00.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:00.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:00.765 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:00.765+0000 7fa5ff0460c0 -1 Falling back to public interface 2021-07-26T07:04:01.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:01.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:01.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:04:01.751 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:04:01.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:01.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:02.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:03.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:03.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:03.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:04:03.048 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:04:03.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:03.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:03.243 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:03.243+0000 7fa5ff0460c0 -1 osd.1 73 log_to_monitors {default=true} 2021-07-26T07:04:03.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:04.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:04.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:04.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:04:04.370 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:04:04.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:04.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:04.643 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [67,73) [v2:127.0.0.1:6810/343419,v1:127.0.0.1:6811/343419] [v2:127.0.0.1:6812/343419,v1:127.0.0.1:6813/343419] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:04:04.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:04:04.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:04:04.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:04:04.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:04:04.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:04:04.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:04:04.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:04:04.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:04:04.645 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:04:04.646 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:04:04.646 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:04:04.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:04:04.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:04:04.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:04:04.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:04:04.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:04:04.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:04:04.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:04:04.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:04:05.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:04:05.007 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:04:05.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:04:05.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:04:05.009 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:04:05.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=309237645317 2021-07-26T07:04:05.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 309237645317 2021-07-26T07:04:05.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-309237645317' 2021-07-26T07:04:05.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:04:05.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:04:05.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=330712481794 2021-07-26T07:04:05.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 330712481794 2021-07-26T07:04:05.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-309237645317 1-330712481794' 2021-07-26T07:04:05.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:04:05.283 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-309237645317 2021-07-26T07:04:05.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:04:05.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:04:05.286 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-309237645317 2021-07-26T07:04:05.286 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:04:05.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=309237645317 2021-07-26T07:04:05.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 309237645317' 2021-07-26T07:04:05.286 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 309237645317 2021-07-26T07:04:05.287 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:04:05.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 309237645316 -lt 309237645317 2021-07-26T07:04:05.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:04:06.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:04:06.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:04:06.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 309237645317 -lt 309237645317 2021-07-26T07:04:06.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:04:06.833 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-330712481794 2021-07-26T07:04:06.834 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:04:06.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:04:06.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-330712481794 2021-07-26T07:04:06.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:04:06.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=330712481794 2021-07-26T07:04:06.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2021-07-26T07:04:06.836 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 330712481794 2021-07-26T07:04:06.837 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:04:07.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 330712481794 -lt 330712481794 2021-07-26T07:04:07.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:04:07.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:07.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:07.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:04:07.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:07.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:07.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:07.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:07.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:07.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:07.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:07.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:04:07.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:07.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:07.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:08.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:04:08.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:04:08.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:04:08.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:04:08.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ4 2021-07-26T07:04:08.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 4 % 2 2021-07-26T07:04:08.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=0 2021-07-26T07:04:08.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:04:08.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2021-07-26T07:04:08.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:04:08.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:04:08.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:04:08.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:04:08.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2021-07-26T07:04:08.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:04:08.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:04:08.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:04:08.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:04:08.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:04:08.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:04:08.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:04:08.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:04:08.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:04:08.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:04:08.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:04:08.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2021-07-26T07:04:08.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:04:08.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:04:08.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:04:08.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:04:08.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:04:08.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2021-07-26T07:04:09.473 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:04:09.473 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:04:09.474 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:04:09.474 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:04:09.474 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:04:09.474 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:04:09.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:04:09.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:04:09.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:04:09.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:04:09.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:04:09.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:04:09.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:04:09.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:04:09.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:04:09.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:04:09.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:04:09.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:04:09.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:04:09.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:04:09.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:04:09.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:04:09.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:04:09.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:04:09.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:04:09.998 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:04:09.998 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:09.998 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:04:09.999 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:04:09.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:04:09.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:04:09.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:04:10.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:04:10.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:04:10.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:04:10.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:10.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:04:10.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:04:10.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:04:10.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:04:10.002 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:04:10.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:10.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:04:10.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:04:10.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:04:10.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:04:10.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:04:10.018 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:10.017+0000 7fe3be9300c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:10.020 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:10.020+0000 7fe3be9300c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:10.021 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:10.021+0000 7fe3be9300c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:10.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:04:10.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:04:10.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:04:10.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:04:10.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:04:10.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:10.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:04:10.315 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:04:10.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:10.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:10.536 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:10.536+0000 7fe3be9300c0 -1 Falling back to public interface 2021-07-26T07:04:10.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:11.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:11.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:11.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:04:11.581 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:04:11.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:11.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:11.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:12.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:12.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:12.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:04:12.881 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:04:12.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:12.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:13.021 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:13.021+0000 7fe3be9300c0 -1 osd.0 78 log_to_monitors {default=true} 2021-07-26T07:04:13.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:14.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:14.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:14.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:04:14.187 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:04:14.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:14.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:14.495 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [72,78) [v2:127.0.0.1:6802/344716,v1:127.0.0.1:6803/344716] [v2:127.0.0.1:6804/344716,v1:127.0.0.1:6805/344716] exists,up 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:04:14.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:04:14.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:04:14.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:04:14.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:04:14.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:04:14.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:04:14.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:04:14.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:04:14.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:04:14.498 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:04:14.498 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:04:14.498 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:04:14.498 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:04:14.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:04:14.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:04:14.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:04:14.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:04:14.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:04:14.593 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:04:14.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:04:14.860 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:04:14.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:04:14.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:04:14.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:04:14.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=352187318274 2021-07-26T07:04:14.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 352187318274 2021-07-26T07:04:14.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-352187318274' 2021-07-26T07:04:14.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:04:14.979 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:04:15.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=330712481797 2021-07-26T07:04:15.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 330712481797 2021-07-26T07:04:15.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-352187318274 1-330712481797' 2021-07-26T07:04:15.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:04:15.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-352187318274 2021-07-26T07:04:15.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:04:15.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:04:15.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-352187318274 2021-07-26T07:04:15.100 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:04:15.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=352187318274 2021-07-26T07:04:15.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 352187318274' 2021-07-26T07:04:15.100 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 352187318274 2021-07-26T07:04:15.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:04:15.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 352187318274 2021-07-26T07:04:15.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:04:16.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:04:16.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:04:16.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 352187318274 -lt 352187318274 2021-07-26T07:04:16.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:04:16.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-330712481797 2021-07-26T07:04:16.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:04:16.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:04:16.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-330712481797 2021-07-26T07:04:16.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:04:16.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=330712481797 2021-07-26T07:04:16.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 330712481797' 2021-07-26T07:04:16.644 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 330712481797 2021-07-26T07:04:16.644 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:04:16.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 330712481797 -lt 330712481797 2021-07-26T07:04:16.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:04:16.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:16.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:17.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:04:17.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:17.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:17.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:17.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:17.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:17.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:17.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:17.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:04:17.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:17.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:17.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:17.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:04:17.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:04:17.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:04:17.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:04:17.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ5 2021-07-26T07:04:17.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 5 % 2 2021-07-26T07:04:17.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=1 2021-07-26T07:04:17.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:04:17.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2021-07-26T07:04:17.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:04:17.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:04:17.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:04:17.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:04:17.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2021-07-26T07:04:17.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:04:17.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:04:17.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:04:17.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:04:17.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:04:17.949 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:04:17.949 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:04:17.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:04:17.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:04:17.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:04:18.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:04:18.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2021-07-26T07:04:18.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:04:18.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:04:18.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:04:18.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:04:18.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:04:18.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2021-07-26T07:04:19.288 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:04:19.289 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:04:19.289 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:04:19.289 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:04:19.289 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:04:19.290 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:04:19.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:04:19.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:04:19.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:04:19.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:04:19.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:04:19.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:04:19.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:04:19.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:04:19.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:04:19.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:04:19.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:04:19.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:04:19.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:04:19.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:04:19.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:04:19.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:04:19.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:04:19.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:04:19.814 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:04:19.814 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:04:19.814 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:19.814 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:04:19.815 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:04:19.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:04:19.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:04:19.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:04:19.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:04:19.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:04:19.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:04:19.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:19.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:04:19.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:04:19.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:04:19.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:04:19.818 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:04:19.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:19.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:04:19.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:04:19.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:04:19.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:04:19.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:04:19.833 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:19.832+0000 7fa116d140c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:19.834 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:19.834+0000 7fa116d140c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:19.835 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:19.835+0000 7fa116d140c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:20.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:04:20.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:04:20.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:04:20.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:04:20.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:04:20.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:20.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:04:20.090 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:04:20.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:20.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:20.350 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:20.350+0000 7fa116d140c0 -1 Falling back to public interface 2021-07-26T07:04:20.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:21.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:21.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:21.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:04:21.398 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:04:21.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:21.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:21.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:22.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:22.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:22.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:04:22.673 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:04:22.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:22.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:22.802 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:22.801+0000 7fa116d140c0 -1 osd.1 84 log_to_monitors {default=true} 2021-07-26T07:04:23.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:24.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:24.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:24.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:04:24.060 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:04:24.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:24.060 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:24.368 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6810/346021,v1:127.0.0.1:6811/346021] [v2:127.0.0.1:6812/346021,v1:127.0.0.1:6813/346021] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:04:24.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:04:24.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:04:24.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:04:24.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:04:24.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:04:24.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:04:24.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:04:24.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:04:24.370 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:04:24.370 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:04:24.371 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:04:24.371 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:04:24.371 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:04:24.371 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:04:24.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:04:24.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:04:24.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:04:24.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:04:24.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:04:24.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:04:24.731 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:04:24.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:04:24.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:04:24.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:04:24.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=352187318277 2021-07-26T07:04:25.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 352187318277 2021-07-26T07:04:25.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-352187318277' 2021-07-26T07:04:25.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:04:25.007 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:04:25.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=377957122050 2021-07-26T07:04:25.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 377957122050 2021-07-26T07:04:25.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-352187318277 1-377957122050' 2021-07-26T07:04:25.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:04:25.016 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-352187318277 2021-07-26T07:04:25.017 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:04:25.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:04:25.017 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-352187318277 2021-07-26T07:04:25.018 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:04:25.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=352187318277 2021-07-26T07:04:25.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 352187318277' 2021-07-26T07:04:25.019 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 352187318277 2021-07-26T07:04:25.019 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:04:25.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 352187318276 -lt 352187318277 2021-07-26T07:04:25.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:04:26.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:04:26.272 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:04:26.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 352187318277 -lt 352187318277 2021-07-26T07:04:26.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:04:26.548 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-377957122050 2021-07-26T07:04:26.548 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:04:26.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:04:26.549 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-377957122050 2021-07-26T07:04:26.549 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:04:26.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=377957122050 2021-07-26T07:04:26.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2021-07-26T07:04:26.551 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 377957122050 2021-07-26T07:04:26.551 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:04:26.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 377957122050 -lt 377957122050 2021-07-26T07:04:26.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:04:26.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:26.862 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:27.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:04:27.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:27.231 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:27.231 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:27.231 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:27.232 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:27.232 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:27.232 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:27.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:04:27.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:27.532 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:27.533 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:27.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:04:27.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:04:27.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:04:27.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:04:27.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ6 2021-07-26T07:04:27.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 6 % 2 2021-07-26T07:04:27.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=0 2021-07-26T07:04:27.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:04:27.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: echo extra 2021-07-26T07:04:27.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2021-07-26T07:04:27.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:04:27.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:04:27.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:04:27.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:04:27.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2021-07-26T07:04:27.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:04:27.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:04:27.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:04:27.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:04:27.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:04:27.864 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:04:27.864 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:04:27.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:04:27.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:04:27.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:04:27.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:04:27.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2021-07-26T07:04:27.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:04:27.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:04:27.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:04:27.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:04:27.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:04:27.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2021-07-26T07:04:29.244 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:04:29.244 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:04:29.244 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:04:29.245 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:04:29.245 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:04:29.245 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:04:29.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:04:29.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:04:29.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:04:29.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:04:29.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:04:29.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:04:29.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:04:29.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:04:29.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:04:29.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:04:29.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:04:29.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:04:29.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:04:29.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:04:29.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:04:29.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:04:29.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:04:29.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:04:29.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:04:29.768 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:04:29.768 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:29.769 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:04:29.769 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:04:29.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:04:29.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:04:29.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:04:29.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:04:29.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:04:29.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:04:29.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:29.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:04:29.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:04:29.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:04:29.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:04:29.771 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:04:29.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:29.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:04:29.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:04:29.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:04:29.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:04:29.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:04:29.788 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:29.787+0000 7f3b6e7e50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:29.790 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:29.789+0000 7f3b6e7e50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:29.791 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:29.790+0000 7f3b6e7e50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:30.063 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:04:30.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:04:30.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:04:30.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:04:30.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:04:30.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:30.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:04:30.066 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:04:30.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:30.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:30.311 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:30.310+0000 7f3b6e7e50c0 -1 Falling back to public interface 2021-07-26T07:04:30.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:31.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:31.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:31.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:04:31.375 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:04:31.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:31.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:31.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:32.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:32.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:32.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:04:32.644 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:04:32.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:32.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:32.824 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:32.823+0000 7f3b6e7e50c0 -1 osd.0 89 log_to_monitors {default=true} 2021-07-26T07:04:32.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:33.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:33.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:33.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:04:33.948 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:04:33.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:33.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:34.295 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 93 up_thru 93 down_at 90 last_clean_interval [82,89) [v2:127.0.0.1:6802/347318,v1:127.0.0.1:6803/347318] [v2:127.0.0.1:6804/347318,v1:127.0.0.1:6805/347318] exists,up 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:04:34.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:04:34.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:04:34.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:04:34.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:04:34.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:04:34.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:04:34.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:04:34.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:04:34.298 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:04:34.298 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:04:34.298 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:04:34.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:04:34.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:04:34.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:04:34.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:04:34.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:04:34.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:04:34.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:04:34.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:04:34.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:04:34.669 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:04:34.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:04:34.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:04:34.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:04:34.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=399431958530 2021-07-26T07:04:34.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 399431958530 2021-07-26T07:04:34.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-399431958530' 2021-07-26T07:04:34.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:04:34.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:04:34.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=377957122053 2021-07-26T07:04:34.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 377957122053 2021-07-26T07:04:34.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-399431958530 1-377957122053' 2021-07-26T07:04:34.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:04:34.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-399431958530 2021-07-26T07:04:34.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:04:34.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:04:34.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-399431958530 2021-07-26T07:04:34.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:04:34.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=399431958530 2021-07-26T07:04:34.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 399431958530' 2021-07-26T07:04:34.914 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 399431958530 2021-07-26T07:04:34.914 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:04:35.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 399431958530 2021-07-26T07:04:35.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:04:36.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:04:36.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:04:36.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 399431958530 -lt 399431958530 2021-07-26T07:04:36.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:04:36.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-377957122053 2021-07-26T07:04:36.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:04:36.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:04:36.453 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-377957122053 2021-07-26T07:04:36.453 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:04:36.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=377957122053 2021-07-26T07:04:36.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 377957122053' 2021-07-26T07:04:36.454 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 377957122053 2021-07-26T07:04:36.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:04:36.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 377957122053 -lt 377957122053 2021-07-26T07:04:36.766 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:04:36.766 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:36.767 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:37.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:04:37.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:37.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:37.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:37.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:37.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:37.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:37.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:37.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:04:37.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:37.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:37.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:37.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:04:37.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:04:37.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:04:37.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1082: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2021-07-26T07:04:37.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:04:37.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ7 2021-07-26T07:04:37.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 7 % 2 2021-07-26T07:04:37.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=1 2021-07-26T07:04:37.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:04:37.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n newheader 2021-07-26T07:04:37.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2021-07-26T07:04:37.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:04:37.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:04:37.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:04:37.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:04:37.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2021-07-26T07:04:37.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:04:37.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:04:37.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:04:37.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:04:37.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:04:37.766 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:04:37.767 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:04:37.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:04:37.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:04:37.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:04:37.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:04:37.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2021-07-26T07:04:37.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:04:37.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:04:37.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:04:37.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:04:37.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:04:37.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2021-07-26T07:04:39.141 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:04:39.141 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:04:39.142 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:04:39.142 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:04:39.142 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:04:39.142 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:04:39.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:04:39.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:04:39.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:04:39.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:04:39.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:04:39.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:04:39.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:04:39.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:04:39.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:04:39.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:04:39.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:04:39.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:04:39.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:04:39.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:04:39.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:04:39.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:04:39.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:04:39.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:04:39.665 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:04:39.665 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:04:39.665 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:39.665 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:04:39.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:04:39.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:04:39.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:04:39.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:04:39.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:04:39.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:04:39.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:04:39.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:39.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:04:39.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:04:39.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:04:39.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:04:39.668 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:04:39.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:39.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:04:39.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:04:39.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:04:39.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:04:39.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:04:39.685 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:39.683+0000 7f35d9e690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:39.687 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:39.686+0000 7f35d9e690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:39.688 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:39.687+0000 7f35d9e690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:39.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:04:39.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:04:39.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:04:39.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:04:39.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:04:39.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:39.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:04:39.943 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:04:39.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:39.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:40.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:40.209 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:40.208+0000 7f35d9e690c0 -1 Falling back to public interface 2021-07-26T07:04:41.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:41.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:41.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:04:41.209 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:04:41.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:41.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:41.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:42.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:42.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:42.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:04:42.483 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:04:42.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:42.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:42.737 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:42.736+0000 7f35d9e690c0 -1 osd.1 95 log_to_monitors {default=true} 2021-07-26T07:04:42.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:43.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:43.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:43.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:04:43.755 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:04:43.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:43.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:44.117 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 99 up_thru 99 down_at 96 last_clean_interval [88,95) [v2:127.0.0.1:6810/348616,v1:127.0.0.1:6811/348616] [v2:127.0.0.1:6812/348616,v1:127.0.0.1:6813/348616] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:04:44.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:04:44.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:04:44.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:04:44.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:04:44.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:04:44.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:04:44.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:04:44.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:04:44.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:04:44.120 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:04:44.120 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:04:44.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:04:44.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:04:44.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:04:44.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:04:44.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:04:44.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:04:44.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:04:44.216 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:04:44.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:04:44.501 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:04:44.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:04:44.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:04:44.502 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:04:44.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=399431958533 2021-07-26T07:04:44.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 399431958533 2021-07-26T07:04:44.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-399431958533' 2021-07-26T07:04:44.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:04:44.646 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:04:44.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=425201762306 2021-07-26T07:04:44.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 425201762306 2021-07-26T07:04:44.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-399431958533 1-425201762306' 2021-07-26T07:04:44.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:04:44.765 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-399431958533 2021-07-26T07:04:44.765 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:04:44.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:04:44.766 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-399431958533 2021-07-26T07:04:44.766 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:04:44.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=399431958533 2021-07-26T07:04:44.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 399431958533' 2021-07-26T07:04:44.768 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 399431958533 2021-07-26T07:04:44.768 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:04:45.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 399431958532 -lt 399431958533 2021-07-26T07:04:45.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:04:46.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:04:46.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:04:46.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 399431958533 -lt 399431958533 2021-07-26T07:04:46.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:04:46.324 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-425201762306 2021-07-26T07:04:46.324 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:04:46.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:04:46.325 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-425201762306 2021-07-26T07:04:46.325 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:04:46.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=425201762306 2021-07-26T07:04:46.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 425201762306' 2021-07-26T07:04:46.326 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 425201762306 2021-07-26T07:04:46.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:04:46.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 425201762306 -lt 425201762306 2021-07-26T07:04:46.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:04:46.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:46.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:46.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:04:46.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:46.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:46.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:46.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:46.999 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:46.999 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:46.999 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:47.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:04:47.309 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:47.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:47.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:47.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:04:47.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:04:47.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:04:47.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2021-07-26T07:04:47.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:04:47.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ8 2021-07-26T07:04:47.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 8 % 2 2021-07-26T07:04:47.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=0 2021-07-26T07:04:47.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:04:47.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2021-07-26T07:04:54.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2021-07-26T07:04:54.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: echo -n bad-val 2021-07-26T07:04:54.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1098: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2021-07-26T07:04:54.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:04:54.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:04:54.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:04:54.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:04:54.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2021-07-26T07:04:54.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:04:54.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:04:54.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:04:54.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:04:54.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:04:54.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:04:54.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:04:54.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:04:54.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:04:54.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:04:54.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:04:54.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2021-07-26T07:04:54.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:04:54.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:04:54.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:04:54.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:04:54.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:04:54.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2021-07-26T07:04:55.838 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:04:55.839 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:04:55.839 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:04:55.839 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:04:55.839 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:04:55.840 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:04:56.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:04:56.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:04:56.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:04:56.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:04:56.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:04:56.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:04:56.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:04:56.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:04:56.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:04:56.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:04:56.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:04:56.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:04:56.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:04:56.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:04:56.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:04:56.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:04:56.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:04:56.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:04:56.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:04:56.364 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:04:56.364 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:56.364 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:04:56.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:04:56.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:04:56.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:04:56.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:04:56.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:04:56.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:04:56.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:04:56.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:56.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:04:56.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:04:56.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:04:56.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:04:56.367 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:04:56.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:56.368 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:04:56.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:04:56.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:04:56.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:04:56.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:04:56.383 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:56.383+0000 7ff00b4600c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:56.385 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:56.385+0000 7ff00b4600c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:56.386 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:56.386+0000 7ff00b4600c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:56.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:04:56.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:04:56.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:04:56.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:04:56.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:04:56.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:56.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:04:56.643 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:04:56.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:56.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:56.909 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:56.908+0000 7ff00b4600c0 -1 Falling back to public interface 2021-07-26T07:04:56.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:57.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:57.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:57.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:04:57.914 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:04:57.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:57.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:58.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:59.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:59.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:59.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:04:59.209 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:04:59.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:59.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:59.260 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:04:59.260+0000 7ff00b4600c0 -1 osd.0 101 log_to_monitors {default=true} 2021-07-26T07:04:59.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:00.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:00.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:00.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:05:00.488 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:05:00.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:00.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:05:00.759 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 105 up_thru 105 down_at 102 last_clean_interval [93,101) [v2:127.0.0.1:6802/349954,v1:127.0.0.1:6803/349954] [v2:127.0.0.1:6804/349954,v1:127.0.0.1:6805/349954] exists,up 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:05:00.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:05:00.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:05:00.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:05:00.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:05:00.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:05:00.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:05:00.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:05:00.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:05:00.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:05:00.762 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:05:00.762 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:05:00.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:05:00.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:05:00.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:05:00.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:05:00.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:05:00.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:05:00.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:05:00.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:05:01.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:05:01.121 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:05:01.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:05:01.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:05:01.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:05:01.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=450971566082 2021-07-26T07:05:01.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 450971566082 2021-07-26T07:05:01.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-450971566082' 2021-07-26T07:05:01.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:05:01.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:05:01.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=425201762310 2021-07-26T07:05:01.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 425201762310 2021-07-26T07:05:01.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-450971566082 1-425201762310' 2021-07-26T07:05:01.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:05:01.397 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-450971566082 2021-07-26T07:05:01.398 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:05:01.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:05:01.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-450971566082 2021-07-26T07:05:01.400 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:05:01.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=450971566082 2021-07-26T07:05:01.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 450971566082' 2021-07-26T07:05:01.400 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 450971566082 2021-07-26T07:05:01.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:05:01.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 450971566082 2021-07-26T07:05:01.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:05:02.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:05:02.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:05:02.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 450971566082 -lt 450971566082 2021-07-26T07:05:02.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:05:02.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-425201762310 2021-07-26T07:05:02.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:05:02.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:05:02.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-425201762310 2021-07-26T07:05:02.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:05:02.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=425201762310 2021-07-26T07:05:02.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 425201762310' 2021-07-26T07:05:02.945 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 425201762310 2021-07-26T07:05:02.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:05:03.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 425201762310 -lt 425201762310 2021-07-26T07:05:03.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:05:03.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:05:03.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:05:03.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:05:03.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:05:03.638 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:05:03.638 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:05:03.638 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:05:03.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:05:03.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:05:03.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:05:03.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:05:03.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:05:03.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:05:03.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:05:04.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:05:04.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:05:04.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:05:04.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2021-07-26T07:05:04.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:05:04.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:05:04.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:05:04.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:05:04.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2021-07-26T07:05:04.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:05:04.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:05:04.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:05:04.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:05:04.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:05:04.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:05:04.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:05:04.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:05:04.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:05:04.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:05:04.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:05:04.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2021-07-26T07:05:04.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:05:04.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:05:04.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:05:04.373 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:05:04.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:05:04.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2021-07-26T07:05:05.588 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:05:05.588 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:05:05.588 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:05:05.589 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:05:05.589 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:05:05.589 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:05:06.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:05:06.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:05:06.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:05:06.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:05:06.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:05:06.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:05:06.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:05:06.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:05:06.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:05:06.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:05:06.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:05:06.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:05:06.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:05:06.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:05:06.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:05:06.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:05:06.112 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:05:06.112 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:05:06.112 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:05:06.113 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:05:06.113 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:05:06.113 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:05:06.113 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:05:06.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:05:06.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:05:06.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:05:06.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:05:06.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:05:06.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:05:06.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:05:06.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:05:06.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:05:06.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:05:06.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:05:06.116 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:05:06.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:05:06.116 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:05:06.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:05:06.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:05:06.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:05:06.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:05:06.131 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:06.131+0000 7f70b54fe0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:06.133 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:06.133+0000 7f70b54fe0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:06.134 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:06.134+0000 7f70b54fe0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:06.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:05:06.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:05:06.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:05:06.392 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:05:06.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:05:06.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:06.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:05:06.394 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:05:06.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:06.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:06.652 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:06.651+0000 7f70b54fe0c0 -1 Falling back to public interface 2021-07-26T07:05:06.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:07.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:07.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:07.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:05:07.702 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:05:07.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:07.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:07.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:08.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:08.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:08.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:05:08.982 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:05:08.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:08.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:09.109 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:09.109+0000 7f70b54fe0c0 -1 osd.1 106 log_to_monitors {default=true} 2021-07-26T07:05:09.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:10.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:10.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:10.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:05:10.257 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:05:10.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:10.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:10.659 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 110 up_thru 110 down_at 107 last_clean_interval [99,106) [v2:127.0.0.1:6810/351250,v1:127.0.0.1:6811/351250] [v2:127.0.0.1:6812/351250,v1:127.0.0.1:6813/351250] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:05:10.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:05:10.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:05:10.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:05:10.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:05:10.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:05:10.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:05:10.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:05:10.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:05:10.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:05:10.661 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:05:10.662 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:05:10.662 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:05:10.662 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:05:10.662 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:05:10.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:05:10.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:05:10.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:05:10.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:05:10.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:05:11.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:05:11.046 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:05:11.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:05:11.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:05:11.047 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:05:11.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=450971566085 2021-07-26T07:05:11.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 450971566085 2021-07-26T07:05:11.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-450971566085' 2021-07-26T07:05:11.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:05:11.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:05:11.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=472446402562 2021-07-26T07:05:11.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 472446402562 2021-07-26T07:05:11.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-450971566085 1-472446402562' 2021-07-26T07:05:11.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:05:11.285 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-450971566085 2021-07-26T07:05:11.285 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:05:11.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:05:11.286 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-450971566085 2021-07-26T07:05:11.286 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:05:11.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=450971566085 2021-07-26T07:05:11.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 450971566085' 2021-07-26T07:05:11.287 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 450971566085 2021-07-26T07:05:11.288 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:05:11.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 450971566084 -lt 450971566085 2021-07-26T07:05:11.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:05:12.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:05:12.552 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:05:12.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 450971566085 -lt 450971566085 2021-07-26T07:05:12.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:05:12.834 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-472446402562 2021-07-26T07:05:12.834 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:05:12.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:05:12.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-472446402562 2021-07-26T07:05:12.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:05:12.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=472446402562 2021-07-26T07:05:12.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 472446402562' 2021-07-26T07:05:12.837 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 472446402562 2021-07-26T07:05:12.837 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:05:13.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 472446402562 -lt 472446402562 2021-07-26T07:05:13.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:05:13.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:05:13.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:05:13.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:05:13.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:05:13.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:05:13.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:05:13.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:05:13.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:05:13.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:05:13.460 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:05:13.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:05:13.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:05:13.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:05:13.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:05:14.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:05:14.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:05:14.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:05:14.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1100: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2021-07-26T07:05:14.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1101: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2021-07-26T07:05:14.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:05:14.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:05:14.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:05:14.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:05:14.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2021-07-26T07:05:14.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:05:14.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:05:14.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:05:14.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:05:14.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:05:14.088 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:05:14.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:05:14.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:05:14.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:05:14.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:05:14.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:05:14.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2021-07-26T07:05:14.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:05:14.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:05:14.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:05:14.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:05:14.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:05:14.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2021-07-26T07:05:15.435 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:05:15.436 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:05:15.436 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:05:15.436 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:05:15.436 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:05:15.437 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:05:15.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:05:15.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:05:15.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:05:15.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:05:15.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:05:15.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:05:15.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:05:15.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:05:15.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:05:15.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:05:15.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:05:15.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:05:15.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:05:15.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:05:15.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:05:15.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:05:15.959 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:05:15.959 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:05:15.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:05:15.960 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:05:15.960 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:05:15.960 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:05:15.960 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:05:15.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:05:15.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:05:15.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:05:15.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:05:15.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:05:15.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:05:15.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:05:15.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:05:15.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:05:15.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:05:15.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:05:15.963 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:05:15.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:05:15.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:05:15.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:05:15.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:05:15.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:05:15.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:05:15.980 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:15.980+0000 7fc5b53620c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:15.982 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:15.982+0000 7fc5b53620c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:15.983 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:15.983+0000 7fc5b53620c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:16.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:05:16.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:05:16.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:05:16.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:05:16.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:05:16.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:16.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:05:16.266 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:05:16.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:16.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:16.500 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:16.499+0000 7fc5b53620c0 -1 Falling back to public interface 2021-07-26T07:05:16.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:17.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:17.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:17.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:05:17.533 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:05:17.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:17.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:17.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:18.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:18.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:18.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:05:18.810 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:05:18.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:18.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:18.918 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:18.917+0000 7fc5b53620c0 -1 osd.1 112 log_to_monitors {default=true} 2021-07-26T07:05:19.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:20.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:20.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:20.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:05:20.189 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:05:20.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:20.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:20.476 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 116 up_thru 116 down_at 113 last_clean_interval [110,112) [v2:127.0.0.1:6810/352547,v1:127.0.0.1:6811/352547] [v2:127.0.0.1:6812/352547,v1:127.0.0.1:6813/352547] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:05:20.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:05:20.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:05:20.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:05:20.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:05:20.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:05:20.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:05:20.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:05:20.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:05:20.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:05:20.478 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:05:20.478 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:05:20.479 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:05:20.479 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:05:20.479 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:05:20.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:05:20.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:05:20.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:05:20.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:05:20.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:05:20.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:05:20.841 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:05:20.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:05:20.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:05:20.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:05:21.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=450971566088 2021-07-26T07:05:21.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 450971566088 2021-07-26T07:05:21.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-450971566088' 2021-07-26T07:05:21.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:05:21.006 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:05:21.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=498216206338 2021-07-26T07:05:21.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 498216206338 2021-07-26T07:05:21.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-450971566088 1-498216206338' 2021-07-26T07:05:21.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:05:21.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-450971566088 2021-07-26T07:05:21.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:05:21.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:05:21.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-450971566088 2021-07-26T07:05:21.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:05:21.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=450971566088 2021-07-26T07:05:21.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 450971566088' 2021-07-26T07:05:21.300 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 450971566088 2021-07-26T07:05:21.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:05:21.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 450971566087 -lt 450971566088 2021-07-26T07:05:21.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:05:22.391 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:05:22.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:05:22.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 450971566088 -lt 450971566088 2021-07-26T07:05:22.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:05:22.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-498216206338 2021-07-26T07:05:22.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:05:22.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:05:22.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-498216206338 2021-07-26T07:05:22.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:05:22.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=498216206338 2021-07-26T07:05:22.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 498216206338' 2021-07-26T07:05:22.672 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 498216206338 2021-07-26T07:05:22.673 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:05:22.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 498216206338 -lt 498216206338 2021-07-26T07:05:22.973 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:05:22.973 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:05:22.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:05:23.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:05:23.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:05:23.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:05:23.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:05:23.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:05:23.337 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:05:23.337 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:05:23.337 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:05:23.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:05:23.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:05:23.603 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:05:23.603 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:05:23.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:05:23.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:05:23.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:05:23.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1102: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2021-07-26T07:05:23.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:05:23.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ9 2021-07-26T07:05:23.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 9 % 2 2021-07-26T07:05:23.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=1 2021-07-26T07:05:23.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:05:23.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2021-07-26T07:05:23.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:05:23.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:05:23.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:05:23.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:05:23.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2021-07-26T07:05:23.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:05:23.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:05:23.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:05:23.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:05:23.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:05:23.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:05:23.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:05:23.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:05:23.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:05:23.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:05:24.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:05:24.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2021-07-26T07:05:24.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:05:24.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:05:24.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:05:24.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:05:24.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:05:24.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2021-07-26T07:05:25.365 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:05:25.365 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:05:25.365 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:05:25.366 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:05:25.366 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:05:25.366 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:05:25.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:05:25.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:05:25.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:05:25.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:05:25.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:05:25.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:05:25.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:05:25.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:05:25.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:05:25.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:05:25.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:05:25.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:05:25.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:05:25.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:05:25.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:05:25.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:05:25.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:05:25.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:05:25.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:05:25.890 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:05:25.890 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:05:25.891 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:05:25.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:05:25.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:05:25.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:05:25.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:05:25.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:05:25.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:05:25.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:05:25.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:05:25.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:05:25.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:05:25.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:05:25.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:05:25.893 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:05:25.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:05:25.894 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:05:25.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:05:25.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:05:25.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:05:25.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:05:25.909 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:25.908+0000 7fb9887c20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:25.911 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:25.910+0000 7fb9887c20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:25.912 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:25.911+0000 7fb9887c20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:26.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:05:26.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:05:26.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:05:26.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:05:26.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:05:26.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:26.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:05:26.167 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:05:26.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:26.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:26.430 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:26.429+0000 7fb9887c20c0 -1 Falling back to public interface 2021-07-26T07:05:26.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:27.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:27.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:27.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:05:27.479 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:05:27.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:27.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:27.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:28.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:28.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:28.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:05:28.757 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:05:28.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:28.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:28.836 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:28.835+0000 7fb9887c20c0 -1 osd.1 117 log_to_monitors {default=true} 2021-07-26T07:05:29.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:30.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:30.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:30.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:05:30.124 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:05:30.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:30.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:30.439 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 121 up_thru 121 down_at 118 last_clean_interval [116,117) [v2:127.0.0.1:6810/353853,v1:127.0.0.1:6811/353853] [v2:127.0.0.1:6812/353853,v1:127.0.0.1:6813/353853] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:05:30.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:05:30.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:05:30.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:05:30.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:05:30.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:05:30.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:05:30.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:05:30.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:05:30.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:05:30.441 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:05:30.442 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:05:30.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:05:30.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:05:30.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:05:30.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:05:30.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:05:30.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:05:30.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:05:30.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:05:30.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:05:30.810 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:05:30.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:05:30.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:05:30.811 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:05:30.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=450971566091 2021-07-26T07:05:30.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 450971566091 2021-07-26T07:05:30.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-450971566091' 2021-07-26T07:05:30.931 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:05:30.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:05:31.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=519691042818 2021-07-26T07:05:31.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 519691042818 2021-07-26T07:05:31.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-450971566091 1-519691042818' 2021-07-26T07:05:31.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:05:31.049 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-450971566091 2021-07-26T07:05:31.049 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:05:31.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:05:31.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-450971566091 2021-07-26T07:05:31.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:05:31.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=450971566091 2021-07-26T07:05:31.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 450971566091' 2021-07-26T07:05:31.052 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 450971566091 2021-07-26T07:05:31.052 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:05:31.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 450971566090 -lt 450971566091 2021-07-26T07:05:31.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:05:32.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:05:32.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:05:32.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 450971566091 -lt 450971566091 2021-07-26T07:05:32.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:05:32.601 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-519691042818 2021-07-26T07:05:32.601 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:05:32.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:05:32.603 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-519691042818 2021-07-26T07:05:32.603 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:05:32.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=519691042818 2021-07-26T07:05:32.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 519691042818' 2021-07-26T07:05:32.604 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 519691042818 2021-07-26T07:05:32.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:05:32.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 519691042818 -lt 519691042818 2021-07-26T07:05:32.871 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:05:32.871 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:05:32.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:05:33.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:05:33.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:05:33.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:05:33.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:05:33.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:05:33.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:05:33.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:05:33.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:05:33.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:05:33.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:05:33.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:05:33.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:05:33.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:05:33.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:05:33.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:05:33.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: echo -n D 2021-07-26T07:05:33.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2021-07-26T07:05:40.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2021-07-26T07:05:40.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:05:40.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:05:40.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:05:40.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:05:40.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2021-07-26T07:05:40.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:05:40.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:05:40.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:05:40.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:05:40.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:05:40.788 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:05:40.788 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:05:40.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:05:40.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:05:40.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:05:40.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:05:40.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2021-07-26T07:05:40.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:05:40.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:05:40.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:05:40.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:05:40.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:05:40.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2021-07-26T07:05:42.140 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:05:42.141 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:05:42.141 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:05:42.141 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:05:42.142 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:05:42.142 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:05:42.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:05:42.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:05:42.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:05:42.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:05:42.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:05:42.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:05:42.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:05:42.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:05:42.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:05:42.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:05:42.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:05:42.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:05:42.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:05:42.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:05:42.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:05:42.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:05:42.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:05:42.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:05:42.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:05:42.669 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:05:42.670 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:05:42.670 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:05:42.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:05:42.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:05:42.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:05:42.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:05:42.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:05:42.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:05:42.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:05:42.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:05:42.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:05:42.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:05:42.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:05:42.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:05:42.673 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:05:42.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:05:42.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:05:42.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:05:42.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:05:42.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:05:42.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:05:42.689 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:42.688+0000 7f4d2209c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:42.692 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:42.690+0000 7f4d2209c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:42.693 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:42.691+0000 7f4d2209c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:42.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:05:42.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:05:42.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:05:42.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:05:42.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:05:42.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:42.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:05:42.963 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:05:42.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:42.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:43.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:43.228 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:43.227+0000 7f4d2209c0c0 -1 Falling back to public interface 2021-07-26T07:05:44.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:44.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:44.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:05:44.231 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:05:44.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:44.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:44.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:45.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:45.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:45.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:05:45.511 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:05:45.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:45.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:45.631 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:45.631+0000 7f4d2209c0c0 -1 osd.1 123 log_to_monitors {default=true} 2021-07-26T07:05:45.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:46.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:46.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:46.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:05:46.880 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:05:46.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:46.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:05:47.187 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 127 up_thru 127 down_at 124 last_clean_interval [121,123) [v2:127.0.0.1:6810/355169,v1:127.0.0.1:6811/355169] [v2:127.0.0.1:6812/355169,v1:127.0.0.1:6813/355169] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:05:47.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:05:47.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:05:47.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:05:47.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:05:47.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:05:47.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:05:47.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:05:47.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:05:47.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:05:47.189 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:05:47.190 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:05:47.191 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:05:47.191 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:05:47.191 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:05:47.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:05:47.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:05:47.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:05:47.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:05:47.286 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:05:47.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:05:47.550 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:05:47.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:05:47.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:05:47.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:05:47.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=450971566095 2021-07-26T07:05:47.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 450971566095 2021-07-26T07:05:47.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-450971566095' 2021-07-26T07:05:47.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:05:47.712 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:05:47.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=545460846594 2021-07-26T07:05:47.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 545460846594 2021-07-26T07:05:47.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-450971566095 1-545460846594' 2021-07-26T07:05:47.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:05:47.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-450971566095 2021-07-26T07:05:47.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:05:47.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:05:47.834 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-450971566095 2021-07-26T07:05:47.834 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:05:47.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=450971566095 2021-07-26T07:05:47.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 450971566095' 2021-07-26T07:05:47.835 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 450971566095 2021-07-26T07:05:47.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:05:48.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 450971566095 -lt 450971566095 2021-07-26T07:05:48.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:05:48.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-545460846594 2021-07-26T07:05:48.101 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:05:48.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:05:48.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-545460846594 2021-07-26T07:05:48.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:05:48.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=545460846594 2021-07-26T07:05:48.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 545460846594' 2021-07-26T07:05:48.104 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 545460846594 2021-07-26T07:05:48.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:05:48.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 545460846594 2021-07-26T07:05:48.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:05:49.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:05:49.430 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:05:49.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 545460846594 2021-07-26T07:05:49.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:05:50.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:05:50.707 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:05:50.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 545460846594 -lt 545460846594 2021-07-26T07:05:50.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:05:50.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:05:50.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:05:51.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:05:51.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:05:51.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:05:51.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:05:51.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:05:51.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:05:51.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:05:51.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:05:51.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:05:51.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:05:51.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:05:51.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:05:51.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:05:51.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:05:51.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:05:51.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1110: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2021-07-26T07:05:51.976 INFO:tasks.workunit.client.0.smithi071.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2021-07-26T07:05:51.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:05:51.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ10 2021-07-26T07:05:51.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 10 % 2 2021-07-26T07:05:51.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=0 2021-07-26T07:05:51.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:05:51.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:05:51.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ11 2021-07-26T07:05:51.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 11 % 2 2021-07-26T07:05:51.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=1 2021-07-26T07:05:51.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:05:51.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:05:51.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ12 2021-07-26T07:05:51.982 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 12 % 2 2021-07-26T07:05:51.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=0 2021-07-26T07:05:51.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:05:51.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:05:51.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ13 2021-07-26T07:05:51.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 13 % 2 2021-07-26T07:05:51.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=1 2021-07-26T07:05:51.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:05:51.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:05:51.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ14 2021-07-26T07:05:51.984 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 14 % 2 2021-07-26T07:05:51.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=0 2021-07-26T07:05:51.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:05:51.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: echo -n bad-val 2021-07-26T07:05:51.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1120: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2021-07-26T07:05:51.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:05:51.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:05:51.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:05:51.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:05:51.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2021-07-26T07:05:51.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:05:51.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:05:51.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:05:51.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:05:51.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:05:51.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:05:51.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:05:51.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:05:51.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:05:51.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:05:52.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:05:52.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2021-07-26T07:05:52.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:05:52.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:05:52.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:05:52.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:05:52.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:05:52.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2021-07-26T07:05:53.316 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:05:53.317 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:05:53.317 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:05:53.317 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:05:53.318 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:05:53.318 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:05:53.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:05:53.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:05:53.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:05:53.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:05:53.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:05:53.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:05:53.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:05:53.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:05:53.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:05:53.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:05:53.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:05:53.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:05:53.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:05:53.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:05:53.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:05:53.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:05:53.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:05:53.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:05:53.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:05:53.843 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:05:53.843 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:05:53.843 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:05:53.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:05:53.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:05:53.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:05:53.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:05:53.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:05:53.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:05:53.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:05:53.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:05:53.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:05:53.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:05:53.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:05:53.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:05:53.846 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:05:53.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:05:53.846 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:05:53.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:05:53.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:05:53.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:05:53.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:05:53.861 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:53.861+0000 7f8d8b9440c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:53.863 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:53.863+0000 7f8d8b9440c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:53.864 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:53.864+0000 7f8d8b9440c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:05:54.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:05:54.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:05:54.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:05:54.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:05:54.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:05:54.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:54.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:05:54.118 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:05:54.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:54.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:05:54.386 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:54.386+0000 7f8d8b9440c0 -1 Falling back to public interface 2021-07-26T07:05:54.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:55.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:55.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:55.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:05:55.422 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:05:55.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:55.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:05:55.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:56.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:56.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:56.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:05:56.700 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:05:56.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:56.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:05:56.797 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:05:56.797+0000 7f8d8b9440c0 -1 osd.0 129 log_to_monitors {default=true} 2021-07-26T07:05:57.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:05:58.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:05:58.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:05:58.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:05:58.072 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:05:58.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:05:58.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:05:58.347 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 133 up_thru 133 down_at 130 last_clean_interval [105,129) [v2:127.0.0.1:6802/356495,v1:127.0.0.1:6803/356495] [v2:127.0.0.1:6804/356495,v1:127.0.0.1:6805/356495] exists,up 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:05:58.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:05:58.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:05:58.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:05:58.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:05:58.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:05:58.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:05:58.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:05:58.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:05:58.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:05:58.350 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:05:58.350 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:05:58.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:05:58.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:05:58.351 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:05:58.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:05:58.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:05:58.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:05:58.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:05:58.445 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:05:58.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:05:58.722 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:05:58.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:05:58.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:05:58.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:05:58.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=571230650370 2021-07-26T07:05:58.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 571230650370 2021-07-26T07:05:58.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-571230650370' 2021-07-26T07:05:58.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:05:58.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:05:59.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=545460846597 2021-07-26T07:05:59.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 545460846597 2021-07-26T07:05:59.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-571230650370 1-545460846597' 2021-07-26T07:05:59.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:05:59.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-571230650370 2021-07-26T07:05:59.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:05:59.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:05:59.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-571230650370 2021-07-26T07:05:59.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:05:59.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=571230650370 2021-07-26T07:05:59.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 571230650370' 2021-07-26T07:05:59.005 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 571230650370 2021-07-26T07:05:59.005 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:05:59.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 571230650370 2021-07-26T07:05:59.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:06:00.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:06:00.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:00.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 571230650370 -lt 571230650370 2021-07-26T07:06:00.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:00.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-545460846597 2021-07-26T07:06:00.545 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:00.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:06:00.547 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-545460846597 2021-07-26T07:06:00.547 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:00.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=545460846597 2021-07-26T07:06:00.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 545460846597' 2021-07-26T07:06:00.548 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 545460846597 2021-07-26T07:06:00.548 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:06:00.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 545460846597 -lt 545460846597 2021-07-26T07:06:00.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:06:00.868 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:06:00.869 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:06:01.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:06:01.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:06:01.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:06:01.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:06:01.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:06:01.237 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:06:01.238 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:06:01.238 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:06:01.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:06:01.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:06:01.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:06:01.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:06:01.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:06:01.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:06:01.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:06:01.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1121: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2021-07-26T07:06:01.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:06:01.867 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:06:01.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:06:01.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:06:01.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2021-07-26T07:06:01.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:06:01.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:06:01.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:06:01.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:06:01.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:06:01.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:06:01.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:06:01.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:06:01.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:06:01.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:06:01.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:06:01.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2021-07-26T07:06:01.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:06:01.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:06:01.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:06:01.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:06:01.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:06:01.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2021-07-26T07:06:03.294 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:06:03.295 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:06:03.295 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:06:03.295 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:06:03.295 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:06:03.295 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:06:03.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:06:03.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:06:03.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:06:03.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:06:03.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:06:03.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:06:03.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:06:03.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:06:03.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:06:03.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:06:03.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:06:03.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:06:03.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:06:03.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:06:03.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:06:03.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:06:03.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:06:03.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:06:03.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:06:03.818 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:06:03.819 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:03.819 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:06:03.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:06:03.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:06:03.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:06:03.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:06:03.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:06:03.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:06:03.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:06:03.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:03.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:06:03.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:06:03.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:06:03.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:06:03.822 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:06:03.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:03.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:06:03.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:06:03.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:06:03.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:06:03.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:06:03.839 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:03.838+0000 7fa08752a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:03.841 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:03.840+0000 7fa08752a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:03.842 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:03.841+0000 7fa08752a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:04.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:06:04.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:06:04.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:06:04.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:06:04.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:06:04.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:04.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:06:04.096 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:06:04.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:04.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:04.361 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:04.361+0000 7fa08752a0c0 -1 Falling back to public interface 2021-07-26T07:06:04.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:05.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:05.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:05.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:06:05.405 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:06:05.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:05.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:05.681 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:06.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:06.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:06.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:06:06.684 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:06:06.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:06.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:06.799 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:06.799+0000 7fa08752a0c0 -1 osd.1 134 log_to_monitors {default=true} 2021-07-26T07:06:06.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:07.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:07.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:07.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:06:07.958 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:06:07.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:07.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:08.449 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 138 up_thru 138 down_at 135 last_clean_interval [127,134) [v2:127.0.0.1:6810/357791,v1:127.0.0.1:6811/357791] [v2:127.0.0.1:6812/357791,v1:127.0.0.1:6813/357791] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:06:08.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:06:08.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:06:08.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:06:08.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:06:08.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:06:08.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:06:08.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:06:08.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:06:08.451 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:06:08.452 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:06:08.452 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:06:08.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:06:08.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:06:08.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:06:08.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:06:08.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:06:08.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:06:08.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:06:08.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:06:08.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:06:08.813 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:06:08.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:06:08.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:08.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:06:08.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=571230650373 2021-07-26T07:06:08.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 571230650373 2021-07-26T07:06:08.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-571230650373' 2021-07-26T07:06:08.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:08.977 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:06:09.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=592705486850 2021-07-26T07:06:09.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 592705486850 2021-07-26T07:06:09.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-571230650373 1-592705486850' 2021-07-26T07:06:09.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:09.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-571230650373 2021-07-26T07:06:09.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:09.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:06:09.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-571230650373 2021-07-26T07:06:09.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:09.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=571230650373 2021-07-26T07:06:09.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 571230650373' 2021-07-26T07:06:09.096 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 571230650373 2021-07-26T07:06:09.097 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:09.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 571230650372 -lt 571230650373 2021-07-26T07:06:09.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:06:10.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:06:10.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:10.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 571230650373 -lt 571230650373 2021-07-26T07:06:10.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:10.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-592705486850 2021-07-26T07:06:10.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:10.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:06:10.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-592705486850 2021-07-26T07:06:10.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:10.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=592705486850 2021-07-26T07:06:10.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 592705486850' 2021-07-26T07:06:10.642 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 592705486850 2021-07-26T07:06:10.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:06:10.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 592705486850 -lt 592705486850 2021-07-26T07:06:10.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:06:10.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:06:10.953 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:06:11.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:06:11.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:06:11.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:06:11.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:06:11.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:06:11.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:06:11.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:06:11.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:06:11.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:06:11.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:06:11.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:06:11.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:06:11.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:06:11.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:06:11.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:06:11.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1122: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2021-07-26T07:06:11.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:06:11.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ15 2021-07-26T07:06:11.915 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 15 % 2 2021-07-26T07:06:11.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=1 2021-07-26T07:06:11.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:06:11.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2021-07-26T07:06:11.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:06:11.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:06:11.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:06:11.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:06:11.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2021-07-26T07:06:11.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:06:11.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:06:11.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:06:11.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:06:11.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:06:11.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:06:11.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:06:11.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:06:11.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:06:11.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:06:12.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:06:12.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2021-07-26T07:06:12.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:06:12.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:06:12.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:06:12.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:06:12.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:06:12.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2021-07-26T07:06:13.200 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:06:13.200 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:06:13.200 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:06:13.201 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:06:13.201 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:06:13.201 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:06:13.201 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2021-07-26T07:06:13.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:06:13.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:06:13.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:06:13.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:06:13.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:06:13.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:06:13.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:06:13.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:06:13.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:06:13.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:06:13.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:06:13.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:06:13.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:06:13.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:06:13.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:06:13.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:06:13.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:06:13.724 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:06:13.724 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:06:13.724 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:06:13.724 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:13.725 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:06:13.725 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:06:13.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:06:13.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:06:13.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:06:13.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:06:13.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:06:13.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:06:13.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:13.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:06:13.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:06:13.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:06:13.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:06:13.727 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:06:13.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:13.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:06:13.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:06:13.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:06:13.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:06:13.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:06:13.745 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:13.744+0000 7f619fb0a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:13.747 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:13.747+0000 7f619fb0a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:13.748 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:13.748+0000 7f619fb0a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:14.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:06:14.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:06:14.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:06:14.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:06:14.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:06:14.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:14.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:06:14.002 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:06:14.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:14.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:14.264 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:14.264+0000 7f619fb0a0c0 -1 Falling back to public interface 2021-07-26T07:06:14.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:15.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:15.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:15.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:06:15.333 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:06:15.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:15.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:15.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:16.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:16.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:16.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:06:16.605 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:06:16.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:16.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:16.682 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:16.681+0000 7f619fb0a0c0 -1 osd.1 140 log_to_monitors {default=true} 2021-07-26T07:06:16.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:17.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:17.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:17.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:06:17.963 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:06:17.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:17.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:18.259 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 144 up_thru 144 down_at 141 last_clean_interval [138,140) [v2:127.0.0.1:6810/359090,v1:127.0.0.1:6811/359090] [v2:127.0.0.1:6812/359090,v1:127.0.0.1:6813/359090] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:06:18.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:06:18.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:06:18.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:06:18.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:06:18.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:06:18.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:06:18.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:06:18.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:06:18.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:06:18.261 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:06:18.261 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:06:18.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:06:18.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:06:18.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:06:18.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:06:18.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:06:18.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:06:18.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:06:18.356 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:06:18.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:06:18.626 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:06:18.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:06:18.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:18.626 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:06:18.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=571230650376 2021-07-26T07:06:18.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 571230650376 2021-07-26T07:06:18.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-571230650376' 2021-07-26T07:06:18.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:18.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:06:18.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=618475290626 2021-07-26T07:06:18.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 618475290626 2021-07-26T07:06:18.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-571230650376 1-618475290626' 2021-07-26T07:06:18.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:18.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-571230650376 2021-07-26T07:06:18.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:18.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:06:18.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-571230650376 2021-07-26T07:06:18.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:18.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=571230650376 2021-07-26T07:06:18.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 571230650376' 2021-07-26T07:06:18.910 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 571230650376 2021-07-26T07:06:18.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:19.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 571230650375 -lt 571230650376 2021-07-26T07:06:19.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:06:20.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:06:20.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:20.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 571230650376 -lt 571230650376 2021-07-26T07:06:20.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:20.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-618475290626 2021-07-26T07:06:20.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:20.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:06:20.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-618475290626 2021-07-26T07:06:20.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:20.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=618475290626 2021-07-26T07:06:20.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2021-07-26T07:06:20.458 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 618475290626 2021-07-26T07:06:20.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:06:20.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 618475290626 -lt 618475290626 2021-07-26T07:06:20.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:06:20.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:06:20.762 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:06:21.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:06:21.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:06:21.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:06:21.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:06:21.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:06:21.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:06:21.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:06:21.123 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:06:21.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:06:21.424 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:06:21.424 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:06:21.425 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:06:21.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:06:21.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:06:21.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:06:21.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:06:21.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ16 2021-07-26T07:06:21.746 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 16 % 2 2021-07-26T07:06:21.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=0 2021-07-26T07:06:21.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:06:21.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1130: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2021-07-26T07:06:21.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:06:21.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:06:21.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:06:21.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:06:21.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2021-07-26T07:06:21.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:06:21.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:06:21.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:06:21.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:06:21.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:06:21.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:06:21.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:06:21.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:06:21.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:06:21.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:06:21.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:06:21.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2021-07-26T07:06:21.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:06:21.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:06:21.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:06:21.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:06:21.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:06:21.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2021-07-26T07:06:23.055 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:06:23.056 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:06:23.056 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:06:23.056 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:06:23.056 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:06:23.057 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:06:23.057 INFO:tasks.workunit.client.0.smithi071.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2021-07-26T07:06:23.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:06:23.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:06:23.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:06:23.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:06:23.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:06:23.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:06:23.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:06:23.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:06:23.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:06:23.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:06:23.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:06:23.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:06:23.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:06:23.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:06:23.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:06:23.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:06:23.580 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:06:23.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:06:23.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:06:23.581 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:06:23.581 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:23.582 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:06:23.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:06:23.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:06:23.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:06:23.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:06:23.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:06:23.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:06:23.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:06:23.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:23.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:06:23.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:06:23.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:06:23.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:06:23.585 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:06:23.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:23.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:06:23.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:06:23.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:06:23.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:06:23.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:06:23.600 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:23.599+0000 7ff6e7c740c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:23.602 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:23.601+0000 7ff6e7c740c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:23.603 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:23.602+0000 7ff6e7c740c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:23.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:06:23.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:06:23.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:06:23.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:06:23.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:06:23.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:23.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:06:23.856 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:06:23.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:23.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:06:24.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:24.133 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:24.132+0000 7ff6e7c740c0 -1 Falling back to public interface 2021-07-26T07:06:25.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:25.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:25.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:06:25.123 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:06:25.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:25.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:06:25.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:26.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:26.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:26.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:06:26.404 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:06:26.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:26.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:06:26.553 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:26.551+0000 7ff6e7c740c0 -1 osd.0 145 log_to_monitors {default=true} 2021-07-26T07:06:26.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:27.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:27.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:27.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:06:27.741 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:06:27.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:27.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:06:28.008 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 149 up_thru 149 down_at 146 last_clean_interval [133,145) [v2:127.0.0.1:6802/360395,v1:127.0.0.1:6803/360395] [v2:127.0.0.1:6804/360395,v1:127.0.0.1:6805/360395] exists,up 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:06:28.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:06:28.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:06:28.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:06:28.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:06:28.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:06:28.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:06:28.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:06:28.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:06:28.011 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:06:28.011 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:06:28.011 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:06:28.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:06:28.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:06:28.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:06:28.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:06:28.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:06:28.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:06:28.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:06:28.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:06:28.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:06:28.376 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:06:28.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:06:28.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:28.377 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:06:28.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=639950127106 2021-07-26T07:06:28.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 639950127106 2021-07-26T07:06:28.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-639950127106' 2021-07-26T07:06:28.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:28.500 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:06:28.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=618475290629 2021-07-26T07:06:28.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 618475290629 2021-07-26T07:06:28.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-639950127106 1-618475290629' 2021-07-26T07:06:28.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:28.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-639950127106 2021-07-26T07:06:28.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:28.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:06:28.620 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-639950127106 2021-07-26T07:06:28.620 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:28.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=639950127106 2021-07-26T07:06:28.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2021-07-26T07:06:28.621 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 639950127106 2021-07-26T07:06:28.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:28.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 639950127106 2021-07-26T07:06:28.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:06:29.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:06:29.888 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:30.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 639950127106 -lt 639950127106 2021-07-26T07:06:30.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:30.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-618475290629 2021-07-26T07:06:30.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:30.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:06:30.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-618475290629 2021-07-26T07:06:30.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:30.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=618475290629 2021-07-26T07:06:30.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 618475290629' 2021-07-26T07:06:30.168 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 618475290629 2021-07-26T07:06:30.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:06:30.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 618475290629 -lt 618475290629 2021-07-26T07:06:30.433 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:06:30.433 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:06:30.433 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:06:30.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:06:30.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:06:30.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:06:30.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:06:30.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:06:30.802 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:06:30.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:06:30.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:06:31.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:06:31.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:06:31.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:06:31.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:06:31.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:06:31.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:06:31.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:06:31.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1131: TEST_corrupt_scrub_replicated: echo -n bad-val 2021-07-26T07:06:31.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2021-07-26T07:06:31.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:06:31.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:06:31.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:06:31.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:06:31.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2021-07-26T07:06:31.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:06:31.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:06:31.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:06:31.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:06:31.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:06:31.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:06:31.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:06:31.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:06:31.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:06:31.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:06:31.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:06:31.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2021-07-26T07:06:31.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:06:31.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:06:31.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:06:31.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:06:31.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:06:31.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2021-07-26T07:06:32.890 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:06:32.891 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:06:32.891 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:06:32.891 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:06:32.891 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:06:32.892 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:06:32.892 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2021-07-26T07:06:32.892 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2021-07-26T07:06:33.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:06:33.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:06:33.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:06:33.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:06:33.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:06:33.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:06:33.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:06:33.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:06:33.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:06:33.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:06:33.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:06:33.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:06:33.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:06:33.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:06:33.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:06:33.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:06:33.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:06:33.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:06:33.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:06:33.414 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:06:33.414 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:33.414 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:06:33.415 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:06:33.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:06:33.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:06:33.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:06:33.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:06:33.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:06:33.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:06:33.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:33.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:06:33.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:06:33.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:06:33.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:06:33.417 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:06:33.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:33.418 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:06:33.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:06:33.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:06:33.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:06:33.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:06:33.434 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:33.433+0000 7fe3d5b4d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:33.436 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:33.435+0000 7fe3d5b4d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:33.437 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:33.436+0000 7fe3d5b4d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:33.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:06:33.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:06:33.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:06:33.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:06:33.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:06:33.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:33.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:06:33.695 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:06:33.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:33.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:33.957 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:33.956+0000 7fe3d5b4d0c0 -1 Falling back to public interface 2021-07-26T07:06:34.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:35.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:35.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:35.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:06:35.004 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:06:35.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:35.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:35.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:36.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:36.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:36.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:06:36.280 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:06:36.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:36.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:36.485 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:36.484+0000 7fe3d5b4d0c0 -1 osd.1 150 log_to_monitors {default=true} 2021-07-26T07:06:36.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:37.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:37.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:37.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:06:37.609 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:06:37.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:37.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:37.931 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/361691,v1:127.0.0.1:6811/361691] [v2:127.0.0.1:6812/361691,v1:127.0.0.1:6813/361691] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:06:37.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:06:37.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:06:37.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:06:37.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:06:37.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:06:37.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:06:37.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:06:37.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:06:37.934 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:06:37.934 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:06:37.934 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:06:37.934 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:06:37.934 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:06:37.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:06:38.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:06:38.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:06:38.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:06:38.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:06:38.030 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:06:38.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:06:38.339 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:06:38.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:06:38.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:38.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:06:38.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=639950127109 2021-07-26T07:06:38.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 639950127109 2021-07-26T07:06:38.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-639950127109' 2021-07-26T07:06:38.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:38.485 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:06:38.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=661424963586 2021-07-26T07:06:38.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 661424963586 2021-07-26T07:06:38.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-639950127109 1-661424963586' 2021-07-26T07:06:38.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:38.603 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-639950127109 2021-07-26T07:06:38.603 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:38.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:06:38.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-639950127109 2021-07-26T07:06:38.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:38.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=639950127109 2021-07-26T07:06:38.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 639950127109' 2021-07-26T07:06:38.605 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 639950127109 2021-07-26T07:06:38.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:38.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 639950127108 -lt 639950127109 2021-07-26T07:06:38.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:06:39.881 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:06:39.882 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:40.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 639950127109 -lt 639950127109 2021-07-26T07:06:40.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:40.162 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-661424963586 2021-07-26T07:06:40.162 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:40.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:06:40.163 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-661424963586 2021-07-26T07:06:40.163 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:40.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=661424963586 2021-07-26T07:06:40.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2021-07-26T07:06:40.164 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 661424963586 2021-07-26T07:06:40.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:06:40.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 661424963586 -lt 661424963586 2021-07-26T07:06:40.466 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:06:40.466 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:06:40.466 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:06:40.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:06:40.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:06:40.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:06:40.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:06:40.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:06:40.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:06:40.836 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:06:40.836 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:06:41.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:06:41.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:06:41.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:06:41.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:06:41.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:06:41.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:06:41.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:06:41.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:06:41.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ17 2021-07-26T07:06:41.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 17 % 2 2021-07-26T07:06:41.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=1 2021-07-26T07:06:41.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:06:41.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2021-07-26T07:06:41.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1138: TEST_corrupt_scrub_replicated: echo ROBJ17 2021-07-26T07:06:41.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1139: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2021-07-26T07:06:41.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:06:41.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:06:41.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:06:41.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:06:41.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2021-07-26T07:06:41.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:06:41.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:06:41.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:06:41.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:06:41.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:06:41.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:06:41.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:06:41.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:06:41.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:06:41.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:06:41.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:06:41.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2021-07-26T07:06:41.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:06:41.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:06:41.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:06:41.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:06:41.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:06:41.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2021-07-26T07:06:42.865 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:06:42.865 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:06:42.865 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:06:42.866 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:06:42.866 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:06:42.866 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:06:42.866 INFO:tasks.workunit.client.0.smithi071.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2021-07-26T07:06:43.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:06:43.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:06:43.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:06:43.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:06:43.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:06:43.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:06:43.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:06:43.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:06:43.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:06:43.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:06:43.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:06:43.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:06:43.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:06:43.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:06:43.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:06:43.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:06:43.450 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:06:43.450 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:06:43.451 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:06:43.451 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:06:43.451 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:43.451 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:06:43.451 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:06:43.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:06:43.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:06:43.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:06:43.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:06:43.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:06:43.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:06:43.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:43.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:06:43.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:06:43.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:06:43.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:06:43.455 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:06:43.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:43.456 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:06:43.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:06:43.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:06:43.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:06:43.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:06:43.469 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:43.469+0000 7fa599d370c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:43.471 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:43.471+0000 7fa599d370c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:43.472 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:43.472+0000 7fa599d370c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:43.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:06:43.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:06:43.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:06:43.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:06:43.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:06:43.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:43.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:06:43.747 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:06:43.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:43.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:06:43.995 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:43.995+0000 7fa599d370c0 -1 Falling back to public interface 2021-07-26T07:06:44.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:45.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:45.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:45.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:06:45.079 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:06:45.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:45.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:06:45.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:46.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:46.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:46.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:06:46.360 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:06:46.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:46.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:06:46.508 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:46.507+0000 7fa599d370c0 -1 osd.0 156 log_to_monitors {default=true} 2021-07-26T07:06:46.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:47.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:47.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:47.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:06:47.638 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:06:47.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:47.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:06:47.925 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 160 up_thru 160 down_at 157 last_clean_interval [149,156) [v2:127.0.0.1:6802/362988,v1:127.0.0.1:6803/362988] [v2:127.0.0.1:6804/362988,v1:127.0.0.1:6805/362988] exists,up 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:06:47.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:06:47.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:06:47.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:06:47.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:06:47.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:06:47.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:06:47.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:06:47.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:06:47.927 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:06:47.927 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:06:47.928 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:06:47.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:06:47.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:06:47.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:06:48.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:06:48.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:06:48.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:06:48.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:06:48.023 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:06:48.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:06:48.288 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:06:48.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:06:48.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:48.289 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:06:48.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=687194767362 2021-07-26T07:06:48.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 687194767362 2021-07-26T07:06:48.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-687194767362' 2021-07-26T07:06:48.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:48.408 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:06:48.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=661424963589 2021-07-26T07:06:48.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 661424963589 2021-07-26T07:06:48.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-687194767362 1-661424963589' 2021-07-26T07:06:48.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:48.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-687194767362 2021-07-26T07:06:48.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:48.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:06:48.530 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-687194767362 2021-07-26T07:06:48.530 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:48.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=687194767362 2021-07-26T07:06:48.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 687194767362' 2021-07-26T07:06:48.531 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 687194767362 2021-07-26T07:06:48.531 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:48.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 687194767362 2021-07-26T07:06:48.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:06:49.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:06:49.797 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:50.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 687194767362 -lt 687194767362 2021-07-26T07:06:50.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:50.078 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-661424963589 2021-07-26T07:06:50.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:50.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:06:50.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-661424963589 2021-07-26T07:06:50.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:50.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=661424963589 2021-07-26T07:06:50.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 661424963589' 2021-07-26T07:06:50.081 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 661424963589 2021-07-26T07:06:50.081 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:06:50.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 661424963589 -lt 661424963589 2021-07-26T07:06:50.378 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:06:50.378 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:06:50.378 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:06:50.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:06:50.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:06:50.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:06:50.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:06:50.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:06:50.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:06:50.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:06:50.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:06:51.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:06:51.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:06:51.030 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:06:51.030 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:06:51.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:06:51.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:06:51.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:06:51.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1140: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2021-07-26T07:06:51.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:06:51.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:06:51.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:06:51.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:06:51.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2021-07-26T07:06:51.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:06:51.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:06:51.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:06:51.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:06:51.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:06:51.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:06:51.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:06:51.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:06:51.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:06:51.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:06:51.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:06:51.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2021-07-26T07:06:51.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:06:51.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:06:51.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:06:51.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:06:51.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:06:51.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2021-07-26T07:06:52.769 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:06:52.770 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:06:52.770 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:06:52.770 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:06:52.770 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:06:52.771 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:06:52.771 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2021-07-26T07:06:52.771 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2021-07-26T07:06:53.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:06:53.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:06:53.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:06:53.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:06:53.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:06:53.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:06:53.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:06:53.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:06:53.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:06:53.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:06:53.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:06:53.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:06:53.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:06:53.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:06:53.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:06:53.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:06:53.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:06:53.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:06:53.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:06:53.350 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:06:53.351 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:53.351 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:06:53.351 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:06:53.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:06:53.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:06:53.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:06:53.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:06:53.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:06:53.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:06:53.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:53.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:06:53.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:06:53.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:06:53.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:06:53.354 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:06:53.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:53.354 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:06:53.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:06:53.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:06:53.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:06:53.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:06:53.370 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:53.369+0000 7fa7c3bd10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:53.372 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:53.371+0000 7fa7c3bd10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:53.373 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:53.373+0000 7fa7c3bd10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:53.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:06:53.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:06:53.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:06:53.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:06:53.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:06:53.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:53.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:06:53.630 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:06:53.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:53.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:53.888 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:53.887+0000 7fa7c3bd10c0 -1 Falling back to public interface 2021-07-26T07:06:53.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:54.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:54.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:54.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:06:54.898 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:06:54.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:54.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:55.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:56.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:56.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:56.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:06:56.175 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:06:56.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:56.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:56.272 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:06:56.272+0000 7fa7c3bd10c0 -1 osd.1 161 log_to_monitors {default=true} 2021-07-26T07:06:56.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:57.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:57.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:57.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:06:57.481 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:06:57.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:57.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:57.773 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 165 up_thru 165 down_at 162 last_clean_interval [154,161) [v2:127.0.0.1:6810/364284,v1:127.0.0.1:6811/364284] [v2:127.0.0.1:6812/364284,v1:127.0.0.1:6813/364284] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:06:57.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:06:57.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:06:57.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:06:57.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:06:57.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:06:57.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:06:57.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:06:57.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:06:57.776 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:06:57.776 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:06:57.776 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:06:57.776 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:06:57.776 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:06:57.777 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:06:57.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:06:57.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:06:57.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:06:57.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:06:57.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:06:58.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:06:58.161 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:06:58.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:06:58.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:58.161 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:06:58.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=687194767365 2021-07-26T07:06:58.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 687194767365 2021-07-26T07:06:58.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-687194767365' 2021-07-26T07:06:58.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:58.280 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:06:58.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=708669603842 2021-07-26T07:06:58.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 708669603842 2021-07-26T07:06:58.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-687194767365 1-708669603842' 2021-07-26T07:06:58.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:58.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-687194767365 2021-07-26T07:06:58.399 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:58.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:06:58.400 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-687194767365 2021-07-26T07:06:58.400 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:58.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=687194767365 2021-07-26T07:06:58.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 687194767365' 2021-07-26T07:06:58.401 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 687194767365 2021-07-26T07:06:58.401 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:58.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 687194767364 -lt 687194767365 2021-07-26T07:06:58.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:06:59.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:06:59.667 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:59.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 687194767365 -lt 687194767365 2021-07-26T07:06:59.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:59.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-708669603842 2021-07-26T07:06:59.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:59.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:06:59.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-708669603842 2021-07-26T07:06:59.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:59.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=708669603842 2021-07-26T07:06:59.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 708669603842' 2021-07-26T07:06:59.950 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 708669603842 2021-07-26T07:06:59.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:07:00.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 708669603842 -lt 708669603842 2021-07-26T07:07:00.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:07:00.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:00.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:00.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:07:00.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:07:00.567 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:07:00.567 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:07:00.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:07:00.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:07:00.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:07:00.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:07:00.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:07:00.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:07:00.831 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:00.831 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:01.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:07:01.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:07:01.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:07:01.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:07:01.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ18 2021-07-26T07:07:01.156 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 18 % 2 2021-07-26T07:07:01.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=0 2021-07-26T07:07:01.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:07:01.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1145: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2021-07-26T07:07:01.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: echo ROBJ18 2021-07-26T07:07:01.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1147: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2021-07-26T07:07:01.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:07:01.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:07:01.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:07:01.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:07:01.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2021-07-26T07:07:01.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:07:01.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:07:01.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:07:01.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:07:01.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:07:01.162 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:07:01.162 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:07:01.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:07:01.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:07:01.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:07:01.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:07:01.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2021-07-26T07:07:01.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:07:01.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:07:01.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:07:01.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:07:01.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:07:01.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2021-07-26T07:07:02.627 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:07:02.628 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:07:02.628 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:07:02.628 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:07:02.628 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:07:02.629 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:07:02.629 INFO:tasks.workunit.client.0.smithi071.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2021-07-26T07:07:03.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:07:03.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:07:03.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:07:03.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:07:03.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:07:03.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:07:03.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:07:03.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:07:03.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:07:03.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:07:03.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:07:03.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:07:03.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:07:03.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:07:03.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:07:03.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:07:03.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:07:03.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:07:03.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:07:03.221 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:07:03.221 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:03.222 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:07:03.222 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:07:03.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:07:03.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:07:03.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:07:03.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:07:03.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:07:03.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:07:03.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:03.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:07:03.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:07:03.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:07:03.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:07:03.225 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:07:03.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:03.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:07:03.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:07:03.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:07:03.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:07:03.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:07:03.238 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:03.238+0000 7fad33a060c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:03.240 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:03.240+0000 7fad33a060c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:03.241 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:03.241+0000 7fad33a060c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:03.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:07:03.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:07:03.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:07:03.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:07:03.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:07:03.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:03.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:07:03.493 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:07:03.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:03.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:07:03.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:03.772 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:03.772+0000 7fad33a060c0 -1 Falling back to public interface 2021-07-26T07:07:04.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:04.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:04.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:07:04.759 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:07:04.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:04.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:07:05.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:06.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:06.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:06.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:07:06.038 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:07:06.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:06.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:07:06.239 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:06.238+0000 7fad33a060c0 -1 osd.0 166 log_to_monitors {default=true} 2021-07-26T07:07:06.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:07.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:07.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:07.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:07:07.320 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:07:07.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:07.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:07:07.658 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 170 up_thru 170 down_at 167 last_clean_interval [160,166) [v2:127.0.0.1:6802/365581,v1:127.0.0.1:6803/365581] [v2:127.0.0.1:6804/365581,v1:127.0.0.1:6805/365581] exists,up 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:07:07.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:07:07.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:07:07.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:07:07.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:07:07.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:07:07.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:07:07.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:07:07.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:07:07.660 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:07:07.660 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:07:07.660 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:07:07.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:07:07.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:07:07.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:07:07.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:07:07.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:07:07.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:07:07.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:07:07.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:07:08.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:07:08.037 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:07:08.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:07:08.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:08.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:07:08.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=730144440322 2021-07-26T07:07:08.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 730144440322 2021-07-26T07:07:08.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-730144440322' 2021-07-26T07:07:08.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:08.156 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:07:08.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=708669603845 2021-07-26T07:07:08.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 708669603845 2021-07-26T07:07:08.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-730144440322 1-708669603845' 2021-07-26T07:07:08.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:08.287 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-730144440322 2021-07-26T07:07:08.287 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:08.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:07:08.288 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-730144440322 2021-07-26T07:07:08.289 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:08.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=730144440322 2021-07-26T07:07:08.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 730144440322' 2021-07-26T07:07:08.290 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 730144440322 2021-07-26T07:07:08.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:07:08.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 730144440322 2021-07-26T07:07:08.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:07:09.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:07:09.556 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:07:09.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 730144440322 2021-07-26T07:07:09.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:07:10.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:07:10.833 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:07:11.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 730144440322 -lt 730144440322 2021-07-26T07:07:11.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:11.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-708669603845 2021-07-26T07:07:11.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:11.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:07:11.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-708669603845 2021-07-26T07:07:11.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:11.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=708669603845 2021-07-26T07:07:11.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 708669603845' 2021-07-26T07:07:11.107 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 708669603845 2021-07-26T07:07:11.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:07:11.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 708669603845 -lt 708669603845 2021-07-26T07:07:11.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:07:11.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:11.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:11.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:07:11.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:07:11.737 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:07:11.738 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:07:11.738 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:07:11.738 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:07:11.738 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:07:11.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:07:12.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:07:12.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:07:12.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:12.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:12.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:07:12.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:07:12.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:07:12.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2021-07-26T07:07:12.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:07:12.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:07:12.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:07:12.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:07:12.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2021-07-26T07:07:12.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:07:12.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:07:12.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:07:12.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:07:12.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:07:12.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:07:12.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:07:12.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:07:12.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:07:12.372 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:07:12.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:07:12.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2021-07-26T07:07:12.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:07:12.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:07:12.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:07:12.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:07:12.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:07:12.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2021-07-26T07:07:13.645 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:07:13.646 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:07:13.646 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:07:13.646 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:07:13.646 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:07:13.647 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:07:13.647 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2021-07-26T07:07:13.647 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2021-07-26T07:07:14.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:07:14.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:07:14.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:07:14.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:07:14.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:07:14.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:07:14.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:07:14.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:07:14.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:07:14.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:07:14.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:07:14.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:07:14.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:07:14.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:07:14.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:07:14.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:07:14.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:07:14.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:07:14.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:07:14.203 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:07:14.203 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:14.203 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:07:14.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:07:14.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:07:14.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:07:14.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:07:14.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:07:14.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:07:14.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:07:14.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:14.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:07:14.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:07:14.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:07:14.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:07:14.206 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:07:14.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:14.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:07:14.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:07:14.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:07:14.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:07:14.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:07:14.222 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:14.222+0000 7fdfbad940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:14.224 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:14.224+0000 7fdfbad940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:14.225 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:14.225+0000 7fdfbad940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:14.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:07:14.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:07:14.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:07:14.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:07:14.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:07:14.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:14.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:07:14.509 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:07:14.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:14.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:07:14.742 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:14.742+0000 7fdfbad940c0 -1 Falling back to public interface 2021-07-26T07:07:14.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:15.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:15.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:15.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:07:15.779 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:07:15.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:15.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:07:16.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:17.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:17.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:17.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:07:17.089 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:07:17.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:17.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:07:17.090 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:17.089+0000 7fdfbad940c0 -1 osd.1 171 log_to_monitors {default=true} 2021-07-26T07:07:17.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:18.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:18.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:18.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:07:18.472 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:07:18.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:18.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:07:18.783 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 175 up_thru 175 down_at 172 last_clean_interval [165,171) [v2:127.0.0.1:6810/366901,v1:127.0.0.1:6811/366901] [v2:127.0.0.1:6812/366901,v1:127.0.0.1:6813/366901] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:07:18.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:07:18.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:07:18.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:07:18.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:07:18.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:07:18.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:07:18.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:07:18.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:07:18.786 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:07:18.786 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:07:18.787 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:07:18.787 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:07:18.787 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:07:18.788 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:07:18.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:07:18.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:07:18.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:07:18.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:07:18.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:07:19.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:07:19.162 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:07:19.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:07:19.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:19.163 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:07:19.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=730144440325 2021-07-26T07:07:19.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 730144440325 2021-07-26T07:07:19.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-730144440325' 2021-07-26T07:07:19.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:19.323 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:07:19.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=751619276802 2021-07-26T07:07:19.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 751619276802 2021-07-26T07:07:19.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-730144440325 1-751619276802' 2021-07-26T07:07:19.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:19.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-730144440325 2021-07-26T07:07:19.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:19.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:07:19.443 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-730144440325 2021-07-26T07:07:19.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:19.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=730144440325 2021-07-26T07:07:19.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 730144440325' 2021-07-26T07:07:19.444 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 730144440325 2021-07-26T07:07:19.445 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:07:19.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 730144440324 -lt 730144440325 2021-07-26T07:07:19.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:07:20.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:07:20.709 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:07:20.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 730144440325 -lt 730144440325 2021-07-26T07:07:20.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:20.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-751619276802 2021-07-26T07:07:20.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:20.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:07:20.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-751619276802 2021-07-26T07:07:20.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:20.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=751619276802 2021-07-26T07:07:20.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 751619276802' 2021-07-26T07:07:20.991 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 751619276802 2021-07-26T07:07:20.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:07:21.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 751619276802 -lt 751619276802 2021-07-26T07:07:21.298 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:07:21.298 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:21.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:21.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:07:21.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:07:21.663 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:07:21.663 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:07:21.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:07:21.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:07:21.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:07:21.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:07:21.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:07:21.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:07:21.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:21.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:22.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:07:22.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:07:22.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:07:22.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2021-07-26T07:07:22.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:07:22.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:07:22.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:07:22.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:07:22.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2021-07-26T07:07:22.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:07:22.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:07:22.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:07:22.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:07:22.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:07:22.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:07:22.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:07:22.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:07:22.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:07:22.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:07:22.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:07:22.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2021-07-26T07:07:22.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:07:22.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:07:22.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:07:22.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:07:22.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:07:22.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2021-07-26T07:07:23.605 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:07:23.605 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:07:23.605 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:07:23.606 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:07:23.606 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:07:23.606 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:07:23.607 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2021-07-26T07:07:23.607 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2021-07-26T07:07:24.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:07:24.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:07:24.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:07:24.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:07:24.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:07:24.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:07:24.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:07:24.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:07:24.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:07:24.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:07:24.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:07:24.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:07:24.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:07:24.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:07:24.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:07:24.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:07:24.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:07:24.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:07:24.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:07:24.132 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:07:24.133 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:24.133 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:07:24.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:07:24.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:07:24.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:07:24.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:07:24.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:07:24.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:07:24.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:07:24.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:24.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:07:24.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:07:24.135 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:07:24.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:07:24.136 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:07:24.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:24.137 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:07:24.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:07:24.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:07:24.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:07:24.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:07:24.152 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:24.151+0000 7f2bf32480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:24.155 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:24.153+0000 7f2bf32480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:24.156 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:24.154+0000 7f2bf32480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:24.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:07:24.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:07:24.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:07:24.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:07:24.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:07:24.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:24.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:07:24.411 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:07:24.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:24.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:07:24.675 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:24.674+0000 7f2bf32480c0 -1 Falling back to public interface 2021-07-26T07:07:24.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:25.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:25.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:25.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:07:25.679 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:07:25.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:25.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:07:25.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:26.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:26.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:26.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:07:26.960 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:07:26.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:26.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:07:27.135 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:27.134+0000 7f2bf32480c0 -1 osd.1 177 log_to_monitors {default=true} 2021-07-26T07:07:27.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:28.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:28.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:28.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:07:28.256 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:07:28.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:28.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:07:28.532 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 181 up_thru 181 down_at 178 last_clean_interval [175,177) [v2:127.0.0.1:6810/368205,v1:127.0.0.1:6811/368205] [v2:127.0.0.1:6812/368205,v1:127.0.0.1:6813/368205] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:07:28.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:07:28.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:07:28.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:07:28.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:07:28.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:07:28.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:07:28.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:07:28.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:07:28.535 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:07:28.535 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:07:28.536 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:07:28.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:07:28.536 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:07:28.537 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:07:28.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:07:28.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:07:28.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:07:28.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:07:28.628 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:07:28.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:07:28.901 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:07:28.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:07:28.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:28.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:07:29.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=730144440328 2021-07-26T07:07:29.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 730144440328 2021-07-26T07:07:29.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-730144440328' 2021-07-26T07:07:29.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:29.087 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:07:29.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=777389080578 2021-07-26T07:07:29.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 777389080578 2021-07-26T07:07:29.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-730144440328 1-777389080578' 2021-07-26T07:07:29.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:29.206 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-730144440328 2021-07-26T07:07:29.206 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:29.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:07:29.207 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-730144440328 2021-07-26T07:07:29.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:29.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=730144440328 2021-07-26T07:07:29.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 730144440328' 2021-07-26T07:07:29.208 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 730144440328 2021-07-26T07:07:29.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:07:29.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 730144440327 -lt 730144440328 2021-07-26T07:07:29.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:07:30.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:07:30.475 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:07:30.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 730144440328 -lt 730144440328 2021-07-26T07:07:30.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:30.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-777389080578 2021-07-26T07:07:30.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:30.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:07:30.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-777389080578 2021-07-26T07:07:30.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:30.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=777389080578 2021-07-26T07:07:30.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 777389080578' 2021-07-26T07:07:30.756 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 777389080578 2021-07-26T07:07:30.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:07:31.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 777389080578 -lt 777389080578 2021-07-26T07:07:31.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:07:31.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:31.056 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:31.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:07:31.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:07:31.380 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:07:31.380 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:07:31.380 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:07:31.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:07:31.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:07:31.381 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:07:31.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:07:31.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:07:31.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:31.685 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:32.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:07:32.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:07:32.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:07:32.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2021-07-26T07:07:32.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1043: TEST_corrupt_scrub_replicated: objname=ROBJ19 2021-07-26T07:07:32.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: expr 19 % 2 2021-07-26T07:07:32.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: local osd=1 2021-07-26T07:07:32.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1048: TEST_corrupt_scrub_replicated: case $i in 2021-07-26T07:07:32.013 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2021-07-26T07:07:32.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=csr_pool 2021-07-26T07:07:32.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=ROBJ0 2021-07-26T07:07:32.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map csr_pool ROBJ0 2021-07-26T07:07:32.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:07:32.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: local pg=3.0 2021-07-26T07:07:32.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2021-07-26T07:07:32.453 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0: {} 2021-07-26T07:07:32.471 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1: {} 2021-07-26T07:07:32.471 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0: osd_max_object_size = '1048576' 2021-07-26T07:07:32.472 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1: osd_max_object_size = '1048576' 2021-07-26T07:07:32.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2021-07-26T07:07:32.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local pooltype=rep 2021-07-26T07:07:32.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-07-26T07:07:32.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local which=data 2021-07-26T07:07:32.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-07-26T07:07:32.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local poolname=csr_pool 2021-07-26T07:07:32.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-07-26T07:07:32.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local objname=ROBJ11 2021-07-26T07:07:32.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-07-26T07:07:32.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: local dir=td/osd-scrub-repair 2021-07-26T07:07:32.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: shift 2021-07-26T07:07:32.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local shard_id=0 2021-07-26T07:07:32.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: shift 2021-07-26T07:07:32.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-07-26T07:07:32.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: get_osds csr_pool ROBJ11 2021-07-26T07:07:32.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=csr_pool 2021-07-26T07:07:32.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=ROBJ11 2021-07-26T07:07:32.487 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map csr_pool ROBJ11 2021-07-26T07:07:32.487 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:07:32.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=1 2021-07-26T07:07:32.751 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T07:07:32.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 1 0 2021-07-26T07:07:32.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: local -a initial_osds 2021-07-26T07:07:32.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: local osd_id=1 2021-07-26T07:07:32.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: '[' rep '!=' ec ']' 2021-07-26T07:07:32.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: shard_id= 2021-07-26T07:07:32.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: cat td/osd-scrub-repair/1/type 2021-07-26T07:07:32.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: type=bluestore 2021-07-26T07:07:32.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-07-26T07:07:32.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local daemon=osd 2021-07-26T07:07:32.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: set_config: local id=1 2021-07-26T07:07:32.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: local config=bluestore_debug_inject_read_err 2021-07-26T07:07:32.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: local value=true 2021-07-26T07:07:32.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: jq 'has("success")' 2021-07-26T07:07:32.756 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: get_asok_path osd.1 2021-07-26T07:07:32.757 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T07:07:32.757 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T07:07:32.757 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:07:32.757 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:32.757 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:07:32.758 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.1.asok 2021-07-26T07:07:32.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.128899/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-07-26T07:07:32.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: test true == true 2021-07-26T07:07:32.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: inject_eio: local loop=0 2021-07-26T07:07:32.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: inject_eio: grep -q Invalid 2021-07-26T07:07:32.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: get_asok_path osd.1 2021-07-26T07:07:32.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T07:07:32.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T07:07:32.859 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:07:32.859 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:32.860 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:07:32.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.1.asok 2021-07-26T07:07:32.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: CEPH_ARGS= 2021-07-26T07:07:32.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: ceph --admin-daemon /tmp/ceph-asok.128899/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2021-07-26T07:07:32.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2021-07-26T07:07:32.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local pooltype=rep 2021-07-26T07:07:32.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-07-26T07:07:32.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local which=mdata 2021-07-26T07:07:32.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-07-26T07:07:32.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local poolname=csr_pool 2021-07-26T07:07:32.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-07-26T07:07:32.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local objname=ROBJ12 2021-07-26T07:07:32.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-07-26T07:07:32.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: local dir=td/osd-scrub-repair 2021-07-26T07:07:32.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: shift 2021-07-26T07:07:32.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local shard_id=1 2021-07-26T07:07:32.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: shift 2021-07-26T07:07:32.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-07-26T07:07:32.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: get_osds csr_pool ROBJ12 2021-07-26T07:07:32.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=csr_pool 2021-07-26T07:07:32.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=ROBJ12 2021-07-26T07:07:32.969 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map csr_pool ROBJ12 2021-07-26T07:07:32.969 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:07:33.233 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=1 2021-07-26T07:07:33.233 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T07:07:33.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 1 0 2021-07-26T07:07:33.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: local -a initial_osds 2021-07-26T07:07:33.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: local osd_id=0 2021-07-26T07:07:33.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: '[' rep '!=' ec ']' 2021-07-26T07:07:33.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: shard_id= 2021-07-26T07:07:33.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: cat td/osd-scrub-repair/0/type 2021-07-26T07:07:33.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: type=bluestore 2021-07-26T07:07:33.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2021-07-26T07:07:33.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local daemon=osd 2021-07-26T07:07:33.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: set_config: local id=0 2021-07-26T07:07:33.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: local config=bluestore_debug_inject_read_err 2021-07-26T07:07:33.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: local value=true 2021-07-26T07:07:33.238 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: jq 'has("success")' 2021-07-26T07:07:33.239 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: get_asok_path osd.0 2021-07-26T07:07:33.239 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T07:07:33.239 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T07:07:33.239 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:07:33.239 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:33.240 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:07:33.240 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.0.asok 2021-07-26T07:07:33.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.128899/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2021-07-26T07:07:33.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: test true == true 2021-07-26T07:07:33.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: inject_eio: local loop=0 2021-07-26T07:07:33.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: inject_eio: grep -q Invalid 2021-07-26T07:07:33.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: get_asok_path osd.0 2021-07-26T07:07:33.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T07:07:33.341 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T07:07:33.341 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:07:33.341 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:33.341 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:07:33.342 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.0.asok 2021-07-26T07:07:33.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: CEPH_ARGS= 2021-07-26T07:07:33.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: ceph --admin-daemon /tmp/ceph-asok.128899/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2021-07-26T07:07:33.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2021-07-26T07:07:33.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local pooltype=rep 2021-07-26T07:07:33.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-07-26T07:07:33.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local which=mdata 2021-07-26T07:07:33.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-07-26T07:07:33.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local poolname=csr_pool 2021-07-26T07:07:33.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-07-26T07:07:33.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local objname=ROBJ13 2021-07-26T07:07:33.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-07-26T07:07:33.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: local dir=td/osd-scrub-repair 2021-07-26T07:07:33.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: shift 2021-07-26T07:07:33.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local shard_id=1 2021-07-26T07:07:33.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: shift 2021-07-26T07:07:33.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-07-26T07:07:33.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: get_osds csr_pool ROBJ13 2021-07-26T07:07:33.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=csr_pool 2021-07-26T07:07:33.450 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=ROBJ13 2021-07-26T07:07:33.450 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map csr_pool ROBJ13 2021-07-26T07:07:33.450 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:07:33.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=1 2021-07-26T07:07:33.721 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T07:07:33.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 1 0 2021-07-26T07:07:33.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: local -a initial_osds 2021-07-26T07:07:33.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: local osd_id=0 2021-07-26T07:07:33.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: '[' rep '!=' ec ']' 2021-07-26T07:07:33.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: shard_id= 2021-07-26T07:07:33.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: cat td/osd-scrub-repair/0/type 2021-07-26T07:07:33.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: type=bluestore 2021-07-26T07:07:33.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2021-07-26T07:07:33.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local daemon=osd 2021-07-26T07:07:33.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: set_config: local id=0 2021-07-26T07:07:33.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: local config=bluestore_debug_inject_read_err 2021-07-26T07:07:33.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: local value=true 2021-07-26T07:07:33.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: jq 'has("success")' 2021-07-26T07:07:33.729 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: get_asok_path osd.0 2021-07-26T07:07:33.730 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T07:07:33.730 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T07:07:33.730 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:07:33.730 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:33.731 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:07:33.731 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.0.asok 2021-07-26T07:07:33.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.128899/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2021-07-26T07:07:33.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: test true == true 2021-07-26T07:07:33.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: inject_eio: local loop=0 2021-07-26T07:07:33.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: inject_eio: grep -q Invalid 2021-07-26T07:07:33.822 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: get_asok_path osd.0 2021-07-26T07:07:33.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T07:07:33.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T07:07:33.824 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:07:33.824 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:33.824 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:07:33.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.0.asok 2021-07-26T07:07:33.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: CEPH_ARGS= 2021-07-26T07:07:33.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: ceph --admin-daemon /tmp/ceph-asok.128899/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2021-07-26T07:07:33.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2021-07-26T07:07:33.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local pooltype=rep 2021-07-26T07:07:33.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-07-26T07:07:33.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local which=data 2021-07-26T07:07:33.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-07-26T07:07:33.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local poolname=csr_pool 2021-07-26T07:07:33.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-07-26T07:07:33.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local objname=ROBJ13 2021-07-26T07:07:33.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-07-26T07:07:33.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: local dir=td/osd-scrub-repair 2021-07-26T07:07:33.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: shift 2021-07-26T07:07:33.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local shard_id=0 2021-07-26T07:07:33.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: shift 2021-07-26T07:07:33.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-07-26T07:07:33.930 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: get_osds csr_pool ROBJ13 2021-07-26T07:07:33.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=csr_pool 2021-07-26T07:07:33.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=ROBJ13 2021-07-26T07:07:33.931 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map csr_pool ROBJ13 2021-07-26T07:07:33.932 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:07:34.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=1 2021-07-26T07:07:34.195 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T07:07:34.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 1 0 2021-07-26T07:07:34.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: local -a initial_osds 2021-07-26T07:07:34.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: local osd_id=1 2021-07-26T07:07:34.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: '[' rep '!=' ec ']' 2021-07-26T07:07:34.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: shard_id= 2021-07-26T07:07:34.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: cat td/osd-scrub-repair/1/type 2021-07-26T07:07:34.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: type=bluestore 2021-07-26T07:07:34.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-07-26T07:07:34.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local daemon=osd 2021-07-26T07:07:34.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: set_config: local id=1 2021-07-26T07:07:34.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: local config=bluestore_debug_inject_read_err 2021-07-26T07:07:34.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: local value=true 2021-07-26T07:07:34.200 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: jq 'has("success")' 2021-07-26T07:07:34.200 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: get_asok_path osd.1 2021-07-26T07:07:34.200 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T07:07:34.201 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T07:07:34.201 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:07:34.201 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:34.201 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:07:34.201 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.1.asok 2021-07-26T07:07:34.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.128899/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-07-26T07:07:34.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: test true == true 2021-07-26T07:07:34.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: inject_eio: local loop=0 2021-07-26T07:07:34.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: inject_eio: grep -q Invalid 2021-07-26T07:07:34.466 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: get_asok_path osd.1 2021-07-26T07:07:34.466 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T07:07:34.466 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T07:07:34.467 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:07:34.467 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:34.467 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:07:34.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.1.asok 2021-07-26T07:07:34.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: CEPH_ARGS= 2021-07-26T07:07:34.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: ceph --admin-daemon /tmp/ceph-asok.128899/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2021-07-26T07:07:34.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2021-07-26T07:07:34.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: pg_scrub: local pgid=3.0 2021-07-26T07:07:34.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: pg_scrub: get_last_scrub_stamp 3.0 2021-07-26T07:07:34.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:07:34.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:07:34.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:07:34.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:07:34.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: pg_scrub: local last_scrub=2021-07-26T07:03:30.569566+0000 2021-07-26T07:07:34.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: pg_scrub: ceph pg scrub 3.0 2021-07-26T07:07:34.916 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 3.0 on osd.1 to scrub 2021-07-26T07:07:34.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: pg_scrub: wait_for_scrub 3.0 2021-07-26T07:03:30.569566+0000 2021-07-26T07:07:34.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=3.0 2021-07-26T07:07:34.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:03:30.569566+0000 2021-07-26T07:07:34.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:07:34.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:07:34.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:07:34.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:07:34.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:07:34.929 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:07:34.929 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:07:34.929 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:07:35.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:03:30.569566+0000 '>' 2021-07-26T07:03:30.569566+0000 2021-07-26T07:07:35.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:07:36.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:07:36.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:07:36.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:07:36.190 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:07:36.190 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:07:36.190 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:07:36.190 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:07:36.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:03:30.569566+0000 '>' 2021-07-26T07:03:30.569566+0000 2021-07-26T07:07:36.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:07:37.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:07:37.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:07:37.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:07:37.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:07:37.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:07:37.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:07:37.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:07:37.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:03:30.569566+0000 '>' 2021-07-26T07:03:30.569566+0000 2021-07-26T07:07:37.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:07:38.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:07:38.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:07:38.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:07:38.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:07:38.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:07:38.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:07:38.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:07:39.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:07:36.078038+0000 '>' 2021-07-26T07:03:30.569566+0000 2021-07-26T07:07:39.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:07:39.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: ERRORS=0 2021-07-26T07:07:39.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2021-07-26T07:07:39.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2021-07-26T07:07:39.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2021-07-26T07:07:39.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2021-07-26T07:07:39.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2021-07-26T07:07:39.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2021-07-26T07:07:39.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2021-07-26T07:07:39.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2021-07-26T07:07:39.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2021-07-26T07:07:39.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2021-07-26T07:07:39.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1181: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2021-07-26T07:07:39.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2021-07-26T07:07:39.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1183: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2021-07-26T07:07:39.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2021-07-26T07:07:39.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1185: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2021-07-26T07:07:39.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1186: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2021-07-26T07:07:39.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1187: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2021-07-26T07:07:39.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1188: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2021-07-26T07:07:39.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1189: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2021-07-26T07:07:39.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1190: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2021-07-26T07:07:39.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2021-07-26T07:07:39.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1192: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2021-07-26T07:07:39.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2021-07-26T07:07:39.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:07:39.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2021-07-26T07:07:39.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1204: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2021-07-26T07:07:39.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1206: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2021-07-26T07:07:39.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1206: TEST_corrupt_scrub_replicated: test 1 = 1 2021-07-26T07:07:39.207 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1208: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2021-07-26T07:07:39.209 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1208: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2021-07-26T07:07:39.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1210: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2021-07-26T07:07:39.233 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1212: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2021-07-26T07:07:39.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1212: TEST_corrupt_scrub_replicated: epoch=181 2021-07-26T07:07:39.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1214: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2021-07-26T07:07:39.236 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T07:07:39.236 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T07:07:39.236 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T07:07:39.236 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T07:07:39.237 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T07:07:39.237 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T07:07:39.237 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T07:07:39.237 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T07:07:39.238 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T07:07:39.238 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T07:07:39.238 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T07:07:39.238 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2021-07-26T07:07:39.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1214: TEST_corrupt_scrub_replicated: jq .inconsistents 2021-07-26T07:07:39.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1214: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T07:07:39.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2095: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2021-07-26T07:07:39.262 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T07:07:39.262 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T07:07:39.262 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T07:07:39.263 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T07:07:39.263 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T07:07:39.263 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T07:07:39.263 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T07:07:39.264 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T07:07:39.264 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T07:07:39.264 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T07:07:39.264 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T07:07:39.264 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2021-07-26T07:07:39.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2095: TEST_corrupt_scrub_replicated: jq .inconsistents 2021-07-26T07:07:39.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2095: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T07:07:39.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T07:07:39.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T07:07:39.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: test no = yes 2021-07-26T07:07:39.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: test '' = yes 2021-07-26T07:07:39.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: objname=ROBJ9 2021-07-26T07:07:39.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: echo -n ZZZ 2021-07-26T07:07:39.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2021-07-26T07:07:39.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2021-07-26T07:07:39.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:07:39.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:07:39.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:07:39.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:07:39.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2021-07-26T07:07:39.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:07:39.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:07:39.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:07:39.414 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:07:39.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:07:39.417 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:07:39.418 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:07:39.420 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:07:39.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:07:39.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:07:39.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:07:39.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2021-07-26T07:07:39.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:07:39.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:07:39.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:07:39.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:07:39.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:07:39.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2021-07-26T07:07:40.840 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:07:40.840 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:07:40.841 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:07:40.841 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:07:40.841 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:07:40.841 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:07:40.842 INFO:tasks.workunit.client.0.smithi071.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2021-07-26T07:07:41.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:07:41.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:07:41.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:07:41.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:07:41.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:07:41.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:07:41.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:07:41.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:07:41.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:07:41.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:07:41.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:07:41.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:07:41.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:07:41.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:07:41.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:07:41.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:07:41.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:07:41.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:07:41.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:07:41.364 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:07:41.364 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:41.364 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:07:41.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:07:41.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:07:41.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:07:41.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:07:41.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:07:41.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:07:41.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:07:41.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:41.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:07:41.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:07:41.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:07:41.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:07:41.367 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:07:41.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:41.369 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:07:41.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:07:41.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:07:41.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:07:41.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:07:41.384 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:41.383+0000 7f1ab155b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:41.386 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:41.385+0000 7f1ab155b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:41.387 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:41.386+0000 7f1ab155b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:41.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:07:41.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:07:41.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:07:41.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:07:41.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:07:41.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:41.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:07:41.640 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:07:41.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:41.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:07:41.911 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:41.910+0000 7f1ab155b0c0 -1 Falling back to public interface 2021-07-26T07:07:41.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:42.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:42.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:42.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:07:42.944 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:07:42.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:42.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:07:43.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:44.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:44.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:44.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:07:44.224 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:07:44.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:44.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:07:44.573 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:44.572+0000 7f1ab155b0c0 -1 osd.0 183 log_to_monitors {default=true} 2021-07-26T07:07:44.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:45.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:45.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:45.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:07:45.595 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:07:45.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:45.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:07:46.006 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 187 up_thru 187 down_at 184 last_clean_interval [170,183) [v2:127.0.0.1:6802/369952,v1:127.0.0.1:6803/369952] [v2:127.0.0.1:6804/369952,v1:127.0.0.1:6805/369952] exists,up 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:07:46.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:07:46.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:07:46.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:07:46.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:07:46.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:07:46.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:07:46.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:07:46.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:07:46.010 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:07:46.010 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:07:46.010 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:07:46.011 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:07:46.011 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:07:46.011 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:07:46.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:07:46.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:07:46.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:07:46.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:07:46.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:07:46.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:07:46.367 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:07:46.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:07:46.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:46.368 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:07:46.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=803158884354 2021-07-26T07:07:46.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 803158884354 2021-07-26T07:07:46.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-803158884354' 2021-07-26T07:07:46.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:46.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:07:46.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=777389080582 2021-07-26T07:07:46.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 777389080582 2021-07-26T07:07:46.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-803158884354 1-777389080582' 2021-07-26T07:07:46.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:46.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-803158884354 2021-07-26T07:07:46.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:46.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:07:46.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-803158884354 2021-07-26T07:07:46.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:46.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=803158884354 2021-07-26T07:07:46.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 803158884354' 2021-07-26T07:07:46.609 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 803158884354 2021-07-26T07:07:46.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:07:46.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 803158884354 2021-07-26T07:07:46.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:07:47.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:07:47.873 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:07:48.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 803158884354 -lt 803158884354 2021-07-26T07:07:48.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:48.174 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-777389080582 2021-07-26T07:07:48.174 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:48.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:07:48.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-777389080582 2021-07-26T07:07:48.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:48.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=777389080582 2021-07-26T07:07:48.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 777389080582' 2021-07-26T07:07:48.177 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 777389080582 2021-07-26T07:07:48.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:07:48.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 777389080583 -lt 777389080582 2021-07-26T07:07:48.440 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:07:48.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:48.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:48.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:07:48.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:07:48.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:07:48.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:07:48.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:07:48.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:07:48.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:07:48.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:07:49.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:07:49.111 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:07:49.111 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:49.111 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:49.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:07:49.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:07:49.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:07:49.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2113: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2021-07-26T07:07:49.439 INFO:tasks.workunit.client.0.smithi071.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2021-07-26T07:07:49.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: objname=ROBJ10 2021-07-26T07:07:49.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2116: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2021-07-26T07:07:49.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:07:49.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:07:49.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:07:49.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:07:49.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2021-07-26T07:07:49.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:07:49.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:07:49.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:07:49.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:07:49.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:07:49.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:07:49.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:07:49.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:07:49.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:07:49.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:07:49.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:07:49.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2021-07-26T07:07:49.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:07:49.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:07:49.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:07:49.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:07:49.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:07:49.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2021-07-26T07:07:50.974 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:07:50.974 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:07:50.975 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:07:50.975 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:07:50.975 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:07:50.975 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:07:50.975 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2021-07-26T07:07:50.976 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2021-07-26T07:07:51.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:07:51.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:07:51.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:07:51.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:07:51.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:07:51.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:07:51.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:07:51.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:07:51.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:07:51.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:07:51.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:07:51.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:07:51.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:07:51.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:07:51.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:07:51.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:07:51.503 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:07:51.503 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:07:51.503 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:07:51.503 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:07:51.504 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:51.504 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:07:51.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:07:51.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:07:51.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:07:51.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:07:51.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:07:51.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:07:51.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:07:51.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:51.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:07:51.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:07:51.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:07:51.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:07:51.507 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:07:51.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:51.507 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:07:51.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:07:51.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:07:51.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:07:51.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:07:51.518 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:51.518+0000 7f1f43cac0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:51.520 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:51.520+0000 7f1f43cac0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:51.522 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:51.522+0000 7f1f43cac0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:51.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:07:51.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:07:51.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:07:51.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:07:51.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:07:51.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:51.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:07:51.774 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:07:51.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:51.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:07:52.039 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:52.039+0000 7f1f43cac0c0 -1 Falling back to public interface 2021-07-26T07:07:52.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:53.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:53.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:53.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:07:53.103 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:07:53.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:53.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:07:53.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:54.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:54.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:54.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:07:54.382 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:07:54.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:54.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:07:54.603 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:07:54.603+0000 7f1f43cac0c0 -1 osd.1 188 log_to_monitors {default=true} 2021-07-26T07:07:54.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:55.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:55.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:55.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:07:55.662 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:07:55.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:55.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:07:55.936 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 192 up_thru 192 down_at 189 last_clean_interval [181,188) [v2:127.0.0.1:6810/371249,v1:127.0.0.1:6811/371249] [v2:127.0.0.1:6812/371249,v1:127.0.0.1:6813/371249] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:07:55.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:07:55.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:07:55.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:07:55.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:07:55.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:07:55.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:07:55.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:07:55.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:07:55.938 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:07:55.938 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:07:55.938 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:07:55.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:07:55.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:07:55.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:07:56.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:07:56.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:07:56.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:07:56.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:07:56.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:07:56.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:07:56.299 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:07:56.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:07:56.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:56.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:07:56.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=803158884357 2021-07-26T07:07:56.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 803158884357 2021-07-26T07:07:56.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-803158884357' 2021-07-26T07:07:56.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:56.419 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:07:56.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=824633720834 2021-07-26T07:07:56.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 824633720834 2021-07-26T07:07:56.536 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-803158884357 1-824633720834' 2021-07-26T07:07:56.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:56.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-803158884357 2021-07-26T07:07:56.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:56.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:07:56.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-803158884357 2021-07-26T07:07:56.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:56.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=803158884357 2021-07-26T07:07:56.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 803158884357' 2021-07-26T07:07:56.540 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 803158884357 2021-07-26T07:07:56.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:07:56.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 803158884356 -lt 803158884357 2021-07-26T07:07:56.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:07:57.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:07:57.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:07:58.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 803158884357 -lt 803158884357 2021-07-26T07:07:58.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:58.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-824633720834 2021-07-26T07:07:58.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:58.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:07:58.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-824633720834 2021-07-26T07:07:58.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:58.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=824633720834 2021-07-26T07:07:58.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 824633720834' 2021-07-26T07:07:58.096 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 824633720834 2021-07-26T07:07:58.096 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:07:58.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 824633720834 -lt 824633720834 2021-07-26T07:07:58.376 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:07:58.376 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:58.376 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:58.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:07:58.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:07:58.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:07:58.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:07:58.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:07:58.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:07:58.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:07:58.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:07:59.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:07:59.051 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:07:59.051 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:59.051 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:59.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:07:59.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:07:59.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:07:59.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2021-07-26T07:08:06.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2118: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2021-07-26T07:08:06.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:08:06.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:08:06.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:08:06.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:08:06.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2021-07-26T07:08:06.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:08:06.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:08:06.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:08:06.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:08:06.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:08:06.125 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:08:06.125 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:08:06.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:08:06.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:08:06.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:08:06.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:08:06.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2021-07-26T07:08:06.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:08:06.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:08:06.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:08:06.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:08:06.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:08:06.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2021-07-26T07:08:07.446 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:08:07.446 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:08:07.447 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:08:07.447 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:08:07.447 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:08:07.447 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:08:07.448 INFO:tasks.workunit.client.0.smithi071.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2021-07-26T07:08:07.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:08:07.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:08:07.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:08:07.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:08:07.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:08:07.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:08:07.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:08:07.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:08:07.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:08:07.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:08:07.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:08:07.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:08:07.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:08:07.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:08:07.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:08:07.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:08:07.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:08:07.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:08:07.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:08:07.969 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:08:07.969 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:07.970 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:07.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:08:07.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:08:07.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:08:07.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:08:07.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:08:07.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:08:07.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:08:07.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:08:07.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:08:07.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:08:07.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:08:07.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:08:07.972 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:08:07.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:08:07.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:08:07.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:08:07.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:08:07.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:08:07.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:08:07.989 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:07.988+0000 7f8798e240c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:08:07.991 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:07.990+0000 7f8798e240c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:08:07.992 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:07.991+0000 7f8798e240c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:08:08.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:08:08.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:08:08.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:08:08.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:08:08.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:08:08.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:08:08.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:08:08.246 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:08:08.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:08:08.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:08:08.509 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:08.508+0000 7f8798e240c0 -1 Falling back to public interface 2021-07-26T07:08:08.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:08:09.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:08:09.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:08:09.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:08:09.519 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:08:09.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:08:09.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:08:09.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:08:10.711 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:10.711+0000 7f8798e240c0 -1 osd.0 193 log_to_monitors {default=true} 2021-07-26T07:08:10.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:08:10.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:08:10.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:08:10.796 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:08:10.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:08:10.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:08:11.118 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 197 up_thru 192 down_at 194 last_clean_interval [187,193) [v2:127.0.0.1:6802/372565,v1:127.0.0.1:6803/372565] [v2:127.0.0.1:6804/372565,v1:127.0.0.1:6805/372565] exists,up 5e04f48f-0d4c-440e-9f99-bf7e8b923c15 2021-07-26T07:08:11.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:08:11.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:08:11.119 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:08:11.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:08:11.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:08:11.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:08:11.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:08:11.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:08:11.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:08:11.121 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:08:11.121 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:08:11.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:08:11.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:08:11.122 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:08:11.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:08:11.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:08:11.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:08:11.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:08:11.218 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:08:11.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:08:11.605 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:08:11.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:08:11.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:08:11.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:08:11.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=846108557314 2021-07-26T07:08:11.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 846108557314 2021-07-26T07:08:11.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-846108557314' 2021-07-26T07:08:11.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:08:11.807 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:08:11.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=824633720838 2021-07-26T07:08:11.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 824633720838 2021-07-26T07:08:11.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-846108557314 1-824633720838' 2021-07-26T07:08:11.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:08:11.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-846108557314 2021-07-26T07:08:11.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:08:11.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:08:11.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-846108557314 2021-07-26T07:08:11.949 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:08:11.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=846108557314 2021-07-26T07:08:11.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 846108557314' 2021-07-26T07:08:11.950 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 846108557314 2021-07-26T07:08:11.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:08:12.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 846108557314 -lt 846108557314 2021-07-26T07:08:12.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:08:12.239 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-824633720838 2021-07-26T07:08:12.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:08:12.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:08:12.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-824633720838 2021-07-26T07:08:12.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:08:12.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=824633720838 2021-07-26T07:08:12.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 824633720838' 2021-07-26T07:08:12.242 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 824633720838 2021-07-26T07:08:12.243 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:08:12.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 824633720837 -lt 824633720838 2021-07-26T07:08:12.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:08:13.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:08:13.520 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:08:13.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 824633720837 -lt 824633720838 2021-07-26T07:08:13.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:08:14.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:08:14.794 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:08:15.068 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 824633720838 -lt 824633720838 2021-07-26T07:08:15.069 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:08:15.070 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:15.070 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:15.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:08:15.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:15.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:15.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:15.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:15.391 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:15.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:15.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:15.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:08:15.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:15.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:15.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:16.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:08:16.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:08:16.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:08:16.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2021-07-26T07:08:16.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:08:16.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:08:16.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:08:16.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:08:16.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2021-07-26T07:08:16.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:08:16.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:08:16.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:08:16.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:08:16.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:08:16.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:08:16.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:08:16.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:08:16.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:08:16.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:08:16.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:08:16.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2021-07-26T07:08:16.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:08:16.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:08:16.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:08:16.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:08:16.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:08:16.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2021-07-26T07:08:17.484 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:08:17.484 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:08:17.485 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:08:17.485 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:08:17.485 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:08:17.485 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:08:17.486 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2021-07-26T07:08:17.486 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2021-07-26T07:08:18.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:08:18.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:08:18.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:08:18.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:08:18.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:08:18.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:08:18.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:08:18.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:08:18.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:08:18.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:08:18.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:08:18.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:08:18.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:08:18.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:08:18.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:08:18.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:08:18.009 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:08:18.009 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:08:18.009 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:08:18.010 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:08:18.010 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:18.010 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:18.010 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:08:18.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:08:18.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:08:18.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:08:18.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:08:18.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:08:18.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:08:18.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:08:18.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:08:18.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:08:18.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:08:18.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:08:18.013 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:08:18.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:08:18.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:08:18.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:08:18.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:08:18.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:08:18.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:08:18.029 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:18.028+0000 7fee62a170c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:08:18.031 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:18.030+0000 7fee62a170c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:08:18.032 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:18.031+0000 7fee62a170c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:08:18.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:08:18.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:08:18.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:08:18.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:08:18.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:08:18.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:08:18.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:08:18.289 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:08:18.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:08:18.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:08:18.549 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:18.547+0000 7fee62a170c0 -1 Falling back to public interface 2021-07-26T07:08:18.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:08:19.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:08:19.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:08:19.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:08:19.559 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:08:19.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:08:19.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:08:19.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:08:20.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:08:20.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:08:20.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:08:20.829 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:08:20.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:08:20.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:08:20.946 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:20.945+0000 7fee62a170c0 -1 osd.1 198 log_to_monitors {default=true} 2021-07-26T07:08:21.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:08:22.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:08:22.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:08:22.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:08:22.185 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:08:22.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:08:22.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:08:22.502 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 202 up_thru 202 down_at 199 last_clean_interval [192,198) [v2:127.0.0.1:6810/373860,v1:127.0.0.1:6811/373860] [v2:127.0.0.1:6812/373860,v1:127.0.0.1:6813/373860] exists,up f9d715e4-5f23-432c-b848-af796cb0a07b 2021-07-26T07:08:22.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:08:22.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:08:22.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:08:22.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:08:22.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:08:22.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:08:22.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:08:22.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:08:22.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:08:22.505 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:08:22.505 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:08:22.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:08:22.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:08:22.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:08:22.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:08:22.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:08:22.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:08:22.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:08:22.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:08:22.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:08:22.865 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:08:22.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:08:22.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:08:22.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:08:23.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=846108557317 2021-07-26T07:08:23.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 846108557317 2021-07-26T07:08:23.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-846108557317' 2021-07-26T07:08:23.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:08:23.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:08:23.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=867583393794 2021-07-26T07:08:23.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 867583393794 2021-07-26T07:08:23.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-846108557317 1-867583393794' 2021-07-26T07:08:23.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:08:23.144 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-846108557317 2021-07-26T07:08:23.144 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:08:23.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:08:23.145 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-846108557317 2021-07-26T07:08:23.145 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:08:23.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=846108557317 2021-07-26T07:08:23.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 846108557317' 2021-07-26T07:08:23.147 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 846108557317 2021-07-26T07:08:23.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:08:23.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 846108557316 -lt 846108557317 2021-07-26T07:08:23.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:08:24.413 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:08:24.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:08:24.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 846108557317 -lt 846108557317 2021-07-26T07:08:24.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:08:24.681 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-867583393794 2021-07-26T07:08:24.681 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:08:24.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:08:24.682 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-867583393794 2021-07-26T07:08:24.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:08:24.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=867583393794 2021-07-26T07:08:24.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 867583393794' 2021-07-26T07:08:24.684 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 867583393794 2021-07-26T07:08:24.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:08:24.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 867583393794 -lt 867583393794 2021-07-26T07:08:24.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:08:24.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:24.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:25.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:08:25.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:25.357 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:25.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:25.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:25.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:25.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:25.359 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:25.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:08:25.657 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:25.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:25.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:25.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:08:25.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:08:25.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:08:25.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2021-07-26T07:08:25.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2021-07-26T07:08:25.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local pooltype=rep 2021-07-26T07:08:25.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-07-26T07:08:25.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local which=data 2021-07-26T07:08:25.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-07-26T07:08:25.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local poolname=csr_pool 2021-07-26T07:08:25.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-07-26T07:08:25.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local objname=ROBJ11 2021-07-26T07:08:25.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-07-26T07:08:25.989 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: local dir=td/osd-scrub-repair 2021-07-26T07:08:25.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: shift 2021-07-26T07:08:25.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local shard_id=0 2021-07-26T07:08:25.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: shift 2021-07-26T07:08:25.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-07-26T07:08:25.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: get_osds csr_pool ROBJ11 2021-07-26T07:08:25.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=csr_pool 2021-07-26T07:08:25.991 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=ROBJ11 2021-07-26T07:08:25.991 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map csr_pool ROBJ11 2021-07-26T07:08:25.992 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:08:26.254 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=1 2021-07-26T07:08:26.255 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T07:08:26.255 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 1 0 2021-07-26T07:08:26.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: local -a initial_osds 2021-07-26T07:08:26.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: local osd_id=1 2021-07-26T07:08:26.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: '[' rep '!=' ec ']' 2021-07-26T07:08:26.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: shard_id= 2021-07-26T07:08:26.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: cat td/osd-scrub-repair/1/type 2021-07-26T07:08:26.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: type=bluestore 2021-07-26T07:08:26.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-07-26T07:08:26.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local daemon=osd 2021-07-26T07:08:26.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: set_config: local id=1 2021-07-26T07:08:26.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: local config=bluestore_debug_inject_read_err 2021-07-26T07:08:26.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: local value=true 2021-07-26T07:08:26.260 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: jq 'has("success")' 2021-07-26T07:08:26.260 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: get_asok_path osd.1 2021-07-26T07:08:26.260 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T07:08:26.260 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T07:08:26.261 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:08:26.261 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:26.261 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:26.261 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.1.asok 2021-07-26T07:08:26.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.128899/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-07-26T07:08:26.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: test true == true 2021-07-26T07:08:26.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: inject_eio: local loop=0 2021-07-26T07:08:26.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: inject_eio: grep -q Invalid 2021-07-26T07:08:26.362 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: get_asok_path osd.1 2021-07-26T07:08:26.362 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T07:08:26.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T07:08:26.363 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:08:26.363 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:26.363 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:26.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.1.asok 2021-07-26T07:08:26.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: CEPH_ARGS= 2021-07-26T07:08:26.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: ceph --admin-daemon /tmp/ceph-asok.128899/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2021-07-26T07:08:26.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2021-07-26T07:08:26.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local pooltype=rep 2021-07-26T07:08:26.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-07-26T07:08:26.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local which=mdata 2021-07-26T07:08:26.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-07-26T07:08:26.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local poolname=csr_pool 2021-07-26T07:08:26.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-07-26T07:08:26.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local objname=ROBJ12 2021-07-26T07:08:26.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-07-26T07:08:26.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: local dir=td/osd-scrub-repair 2021-07-26T07:08:26.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: shift 2021-07-26T07:08:26.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local shard_id=1 2021-07-26T07:08:26.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: shift 2021-07-26T07:08:26.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-07-26T07:08:26.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: get_osds csr_pool ROBJ12 2021-07-26T07:08:26.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=csr_pool 2021-07-26T07:08:26.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=ROBJ12 2021-07-26T07:08:26.471 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map csr_pool ROBJ12 2021-07-26T07:08:26.471 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:08:26.735 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=1 2021-07-26T07:08:26.735 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T07:08:26.736 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 1 0 2021-07-26T07:08:26.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: local -a initial_osds 2021-07-26T07:08:26.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: local osd_id=0 2021-07-26T07:08:26.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: '[' rep '!=' ec ']' 2021-07-26T07:08:26.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: shard_id= 2021-07-26T07:08:26.737 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: cat td/osd-scrub-repair/0/type 2021-07-26T07:08:26.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: type=bluestore 2021-07-26T07:08:26.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2021-07-26T07:08:26.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local daemon=osd 2021-07-26T07:08:26.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: set_config: local id=0 2021-07-26T07:08:26.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: local config=bluestore_debug_inject_read_err 2021-07-26T07:08:26.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: local value=true 2021-07-26T07:08:26.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: jq 'has("success")' 2021-07-26T07:08:26.741 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: get_asok_path osd.0 2021-07-26T07:08:26.741 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T07:08:26.741 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T07:08:26.741 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:08:26.742 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:26.742 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:26.742 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.0.asok 2021-07-26T07:08:26.742 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.128899/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2021-07-26T07:08:26.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: test true == true 2021-07-26T07:08:26.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: inject_eio: local loop=0 2021-07-26T07:08:26.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: inject_eio: grep -q Invalid 2021-07-26T07:08:26.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: get_asok_path osd.0 2021-07-26T07:08:26.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T07:08:26.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T07:08:26.843 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:08:26.843 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:26.843 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:26.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.0.asok 2021-07-26T07:08:26.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: CEPH_ARGS= 2021-07-26T07:08:26.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: ceph --admin-daemon /tmp/ceph-asok.128899/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2021-07-26T07:08:26.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2021-07-26T07:08:26.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local pooltype=rep 2021-07-26T07:08:26.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-07-26T07:08:26.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local which=mdata 2021-07-26T07:08:26.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-07-26T07:08:26.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local poolname=csr_pool 2021-07-26T07:08:26.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-07-26T07:08:26.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local objname=ROBJ13 2021-07-26T07:08:26.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-07-26T07:08:26.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: local dir=td/osd-scrub-repair 2021-07-26T07:08:26.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: shift 2021-07-26T07:08:26.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local shard_id=1 2021-07-26T07:08:26.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: shift 2021-07-26T07:08:26.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-07-26T07:08:26.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: get_osds csr_pool ROBJ13 2021-07-26T07:08:26.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=csr_pool 2021-07-26T07:08:26.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=ROBJ13 2021-07-26T07:08:26.950 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map csr_pool ROBJ13 2021-07-26T07:08:26.951 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:08:27.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=1 2021-07-26T07:08:27.213 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T07:08:27.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 1 0 2021-07-26T07:08:27.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: local -a initial_osds 2021-07-26T07:08:27.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: local osd_id=0 2021-07-26T07:08:27.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: '[' rep '!=' ec ']' 2021-07-26T07:08:27.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: shard_id= 2021-07-26T07:08:27.214 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: cat td/osd-scrub-repair/0/type 2021-07-26T07:08:27.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: type=bluestore 2021-07-26T07:08:27.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2021-07-26T07:08:27.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local daemon=osd 2021-07-26T07:08:27.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: set_config: local id=0 2021-07-26T07:08:27.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: local config=bluestore_debug_inject_read_err 2021-07-26T07:08:27.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: local value=true 2021-07-26T07:08:27.221 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: jq 'has("success")' 2021-07-26T07:08:27.221 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: get_asok_path osd.0 2021-07-26T07:08:27.221 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T07:08:27.222 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T07:08:27.222 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:08:27.223 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:27.223 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:27.224 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.0.asok 2021-07-26T07:08:27.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.128899/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2021-07-26T07:08:27.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: test true == true 2021-07-26T07:08:27.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: inject_eio: local loop=0 2021-07-26T07:08:27.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: inject_eio: grep -q Invalid 2021-07-26T07:08:27.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: get_asok_path osd.0 2021-07-26T07:08:27.321 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T07:08:27.321 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T07:08:27.321 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:08:27.321 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:27.322 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:27.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.0.asok 2021-07-26T07:08:27.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: CEPH_ARGS= 2021-07-26T07:08:27.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: ceph --admin-daemon /tmp/ceph-asok.128899/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2021-07-26T07:08:27.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2021-07-26T07:08:27.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local pooltype=rep 2021-07-26T07:08:27.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-07-26T07:08:27.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local which=data 2021-07-26T07:08:27.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-07-26T07:08:27.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local poolname=csr_pool 2021-07-26T07:08:27.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-07-26T07:08:27.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local objname=ROBJ13 2021-07-26T07:08:27.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-07-26T07:08:27.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: local dir=td/osd-scrub-repair 2021-07-26T07:08:27.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: shift 2021-07-26T07:08:27.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local shard_id=0 2021-07-26T07:08:27.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: shift 2021-07-26T07:08:27.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-07-26T07:08:27.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: get_osds csr_pool ROBJ13 2021-07-26T07:08:27.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=csr_pool 2021-07-26T07:08:27.430 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=ROBJ13 2021-07-26T07:08:27.430 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map csr_pool ROBJ13 2021-07-26T07:08:27.430 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:08:27.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=1 2021-07-26T07:08:27.695 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T07:08:27.695 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 1 0 2021-07-26T07:08:27.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: local -a initial_osds 2021-07-26T07:08:27.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: local osd_id=1 2021-07-26T07:08:27.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: '[' rep '!=' ec ']' 2021-07-26T07:08:27.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: shard_id= 2021-07-26T07:08:27.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: cat td/osd-scrub-repair/1/type 2021-07-26T07:08:27.696 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: type=bluestore 2021-07-26T07:08:27.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-07-26T07:08:27.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local daemon=osd 2021-07-26T07:08:27.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: set_config: local id=1 2021-07-26T07:08:27.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: local config=bluestore_debug_inject_read_err 2021-07-26T07:08:27.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: local value=true 2021-07-26T07:08:27.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: jq 'has("success")' 2021-07-26T07:08:27.698 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: get_asok_path osd.1 2021-07-26T07:08:27.698 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T07:08:27.699 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T07:08:27.699 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:08:27.699 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:27.699 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:27.700 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.1.asok 2021-07-26T07:08:27.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.128899/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-07-26T07:08:27.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: test true == true 2021-07-26T07:08:27.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: inject_eio: local loop=0 2021-07-26T07:08:27.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: inject_eio: grep -q Invalid 2021-07-26T07:08:27.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: get_asok_path osd.1 2021-07-26T07:08:27.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T07:08:27.800 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T07:08:27.800 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:08:27.800 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:27.800 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:27.801 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-osd.1.asok 2021-07-26T07:08:27.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: CEPH_ARGS= 2021-07-26T07:08:27.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: ceph --admin-daemon /tmp/ceph-asok.128899/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2021-07-26T07:08:27.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2021-07-26T07:08:28.015 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0: {} 2021-07-26T07:08:28.033 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1: {} 2021-07-26T07:08:28.033 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0: osd_max_object_size = '134217728' 2021-07-26T07:08:28.034 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1: osd_max_object_size = '134217728' 2021-07-26T07:08:28.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2021-07-26T07:08:28.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: pg_deep_scrub: local pgid=3.0 2021-07-26T07:08:28.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:08:28.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:28.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:08:28.045 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:28.045 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:08:28.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: pg_deep_scrub: local last_scrub=2021-07-26T07:03:30.569566+0000 2021-07-26T07:08:28.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1800: pg_deep_scrub: ceph pg deep-scrub 3.0 2021-07-26T07:08:28.562 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2021-07-26T07:08:28.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: pg_deep_scrub: wait_for_scrub 3.0 2021-07-26T07:03:30.569566+0000 last_deep_scrub_stamp 2021-07-26T07:08:28.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=3.0 2021-07-26T07:08:28.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:03:30.569566+0000 2021-07-26T07:08:28.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_deep_scrub_stamp 2021-07-26T07:08:28.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:08:28.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:08:28.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:08:28.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:28.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:08:28.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:28.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:08:28.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:03:30.569566+0000 '>' 2021-07-26T07:03:30.569566+0000 2021-07-26T07:08:28.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:08:29.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:08:29.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:08:29.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:08:29.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:29.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:08:29.836 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:29.836 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:08:30.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:03:30.569566+0000 '>' 2021-07-26T07:03:30.569566+0000 2021-07-26T07:08:30.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:08:31.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:08:31.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:08:31.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:08:31.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:31.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:08:31.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:31.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:08:31.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:03:30.569566+0000 '>' 2021-07-26T07:03:30.569566+0000 2021-07-26T07:08:31.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:08:32.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:08:32.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:08:32.380 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:08:32.380 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:32.380 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:08:32.380 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:32.380 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:08:32.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:03:30.569566+0000 '>' 2021-07-26T07:03:30.569566+0000 2021-07-26T07:08:32.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:08:33.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:08:33.647 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:08:33.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:08:33.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:33.649 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:08:33.649 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:33.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:08:33.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:03:30.569566+0000 '>' 2021-07-26T07:03:30.569566+0000 2021-07-26T07:08:33.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:08:34.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:08:34.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:08:34.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:08:34.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:34.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:08:34.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:34.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:08:35.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:03:30.569566+0000 '>' 2021-07-26T07:03:30.569566+0000 2021-07-26T07:08:35.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:08:36.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:08:36.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:08:36.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:08:36.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:36.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:08:36.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:36.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:08:36.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:03:30.569566+0000 '>' 2021-07-26T07:03:30.569566+0000 2021-07-26T07:08:36.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:08:37.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:08:37.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:08:37.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:08:37.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:37.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:08:37.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:37.463 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:08:37.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:03:30.569566+0000 '>' 2021-07-26T07:03:30.569566+0000 2021-07-26T07:08:37.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:08:38.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:08:38.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:08:38.736 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2021-07-26T07:08:38.736 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:38.737 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:08:38.737 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:38.737 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2021-07-26T07:08:39.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:08:31.216775+0000 '>' 2021-07-26T07:03:30.569566+0000 2021-07-26T07:08:39.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:08:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings=() 2021-07-26T07:08:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2021-07-26T07:08:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2021-07-26T07:08:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2021-07-26T07:08:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2021-07-26T07:08:39.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2021-07-26T07:08:39.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2021-07-26T07:08:39.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2021-07-26T07:08:39.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2021-07-26T07:08:39.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2021-07-26T07:08:39.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2021-07-26T07:08:39.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2021-07-26T07:08:39.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2021-07-26T07:08:39.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2021-07-26T07:08:39.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2021-07-26T07:08:39.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2021-07-26T07:08:39.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2021-07-26T07:08:39.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2021-07-26T07:08:39.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2021-07-26T07:08:39.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2021-07-26T07:08:39.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2021-07-26T07:08:39.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2021-07-26T07:08:39.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2021-07-26T07:08:39.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2021-07-26T07:08:39.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2021-07-26T07:08:39.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2021-07-26T07:08:39.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2021-07-26T07:08:39.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2159: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2021-07-26T07:08:39.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2021-07-26T07:08:39.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2161: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2021-07-26T07:08:39.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2021-07-26T07:08:39.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2163: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2021-07-26T07:08:39.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2164: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2021-07-26T07:08:39.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2165: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2021-07-26T07:08:39.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2166: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2021-07-26T07:08:39.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2167: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2021-07-26T07:08:39.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2168: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2021-07-26T07:08:39.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2021-07-26T07:08:39.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2170: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2021-07-26T07:08:39.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2021-07-26T07:08:39.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.080 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.124 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2021-07-26T07:08:39.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2021-07-26T07:08:39.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2182: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2021-07-26T07:08:39.377 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2184: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2021-07-26T07:08:39.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2184: TEST_corrupt_scrub_replicated: test 1 = 1 2021-07-26T07:08:39.379 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2186: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2021-07-26T07:08:39.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2186: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2021-07-26T07:08:39.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2188: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2021-07-26T07:08:39.409 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2190: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2021-07-26T07:08:39.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2190: TEST_corrupt_scrub_replicated: epoch=202 2021-07-26T07:08:39.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2192: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2021-07-26T07:08:39.413 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T07:08:39.413 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T07:08:39.413 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T07:08:39.413 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T07:08:39.414 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T07:08:39.414 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T07:08:39.414 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T07:08:39.414 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T07:08:39.414 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T07:08:39.415 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T07:08:39.415 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T07:08:39.415 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2021-07-26T07:08:39.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2192: TEST_corrupt_scrub_replicated: jq .inconsistents 2021-07-26T07:08:39.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2192: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T07:08:39.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2021-07-26T07:08:39.442 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T07:08:39.442 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T07:08:39.443 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T07:08:39.443 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T07:08:39.443 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T07:08:39.444 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T07:08:39.444 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T07:08:39.444 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T07:08:39.444 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T07:08:39.445 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T07:08:39.445 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T07:08:39.445 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2021-07-26T07:08:39.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: jq .inconsistents 2021-07-26T07:08:39.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T07:08:39.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3572: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T07:08:39.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T07:08:39.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3573: TEST_corrupt_scrub_replicated: test no = yes 2021-07-26T07:08:39.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: test '' = yes 2021-07-26T07:08:39.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3583: TEST_corrupt_scrub_replicated: repair 3.0 2021-07-26T07:08:39.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=3.0 2021-07-26T07:08:39.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 3.0 2021-07-26T07:08:39.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:39.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:08:39.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:39.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:08:39.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T07:08:31.216775+0000 2021-07-26T07:08:39.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 3.0 2021-07-26T07:08:39.983 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 3.0 on osd.1 to repair 2021-07-26T07:08:39.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 3.0 2021-07-26T07:08:31.216775+0000 2021-07-26T07:08:39.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=3.0 2021-07-26T07:08:39.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:08:31.216775+0000 2021-07-26T07:08:39.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:08:39.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:08:39.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:08:39.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:08:39.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:39.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:08:39.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:39.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:08:40.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:08:31.216775+0000 '>' 2021-07-26T07:08:31.216775+0000 2021-07-26T07:08:40.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:08:41.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:08:41.255 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:08:41.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:08:41.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:41.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:08:41.256 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:41.257 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:08:41.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:08:31.216775+0000 '>' 2021-07-26T07:08:31.216775+0000 2021-07-26T07:08:41.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:08:42.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:08:42.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:08:42.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:08:42.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:08:42.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:08:42.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:08:42.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:08:42.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:08:40.989295+0000 '>' 2021-07-26T07:08:31.216775+0000 2021-07-26T07:08:42.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:08:42.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3584: TEST_corrupt_scrub_replicated: wait_for_clean 2021-07-26T07:08:42.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:08:42.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:08:42.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:08:42.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:08:42.848 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:08:42.848 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:08:42.848 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:08:42.849 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:08:42.849 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:08:42.849 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:08:42.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:08:42.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:08:42.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:08:42.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:08:42.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:08:43.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:08:43.211 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:08:43.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:08:43.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:08:43.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:08:43.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=846108557322 2021-07-26T07:08:43.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 846108557322 2021-07-26T07:08:43.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-846108557322' 2021-07-26T07:08:43.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:08:43.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:08:43.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=867583393799 2021-07-26T07:08:43.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 867583393799 2021-07-26T07:08:43.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-846108557322 1-867583393799' 2021-07-26T07:08:43.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:08:43.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-846108557322 2021-07-26T07:08:43.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:08:43.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:08:43.454 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-846108557322 2021-07-26T07:08:43.454 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:08:43.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=846108557322 2021-07-26T07:08:43.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 846108557322' 2021-07-26T07:08:43.455 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 846108557322 2021-07-26T07:08:43.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:08:43.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 846108557321 -lt 846108557322 2021-07-26T07:08:43.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:08:44.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:08:44.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:08:44.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 846108557322 -lt 846108557322 2021-07-26T07:08:44.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:08:45.000 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-867583393799 2021-07-26T07:08:45.000 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:08:45.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:08:45.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-867583393799 2021-07-26T07:08:45.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:08:45.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=867583393799 2021-07-26T07:08:45.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 867583393799' 2021-07-26T07:08:45.003 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 867583393799 2021-07-26T07:08:45.003 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:08:45.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 867583393799 -lt 867583393799 2021-07-26T07:08:45.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:08:45.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:45.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:45.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:08:45.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:45.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:45.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:45.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:45.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:45.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:45.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:45.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:08:45.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:45.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:45.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:46.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:08:46.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:08:46.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:08:46.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3587: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2021-07-26T07:08:46.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2021-07-26T07:08:46.375 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3590: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2021-07-26T07:08:46.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3591: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2021-07-26T07:08:46.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3592: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2021-07-26T07:08:46.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3593: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2021-07-26T07:08:46.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3595: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2021-07-26T07:08:46.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2021-07-26T07:08:46.780 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'csr_pool' removed 2021-07-26T07:08:46.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T07:08:46.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:08:46.791 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:08:46.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:08:46.792 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:08:46.792 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:08:46.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:08:46.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:08:46.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:08:46.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:08:46.898 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:08:46.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:08:46.899 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:08:46.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:08:46.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:08:46.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:08:46.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:08:46.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:08:46.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:08:46.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:08:46.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:08:46.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:08:46.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:08:46.934 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:08:46.934 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:46.934 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:46.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:08:46.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:08:46.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:08:46.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T07:08:46.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T07:08:46.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T07:08:46.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T07:08:46.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:08:46.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:08:46.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:08:46.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:08:46.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:08:46.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:08:46.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:08:46.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:08:46.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:08:46.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:08:46.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:08:46.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:08:46.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:08:46.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:08:46.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:08:46.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:08:46.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:08:46.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:08:46.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:08:46.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:08:46.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:08:46.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:08:46.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:08:46.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:46.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:46.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:08:46.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:08:46.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:08:46.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T07:08:46.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:08:46.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:46.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:46.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T07:08:46.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:08:46.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:08:46.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:08:46.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T07:08:46.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2021-07-26T07:08:46.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2021-07-26T07:08:46.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5944: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2021-07-26T07:08:46.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5945: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2021-07-26T07:08:46.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2021-07-26T07:08:46.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T07:08:46.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:08:46.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:08:46.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:08:46.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T07:08:46.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2021-07-26T07:08:47.392 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:08:47.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:08:47.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:08:47.393 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:08:47.393 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:47.393 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:47.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:08:47.394 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2021-07-26T07:08:47.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:08:47.654 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:08:47.654 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:08:47.655 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:08:47.655 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:08:47.655 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:08:47.655 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:08:47.656 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:08:47.656 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:08:47.656 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:08:47.656 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:47.656 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:47.657 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:08:47.657 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:08:47.657 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T07:08:47.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:08:47.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:08:47.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:08:47.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:08:47.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:08:47.753 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:08:47.753 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:08:47.753 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:08:47.754 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:08:47.754 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:47.754 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:47.754 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:08:47.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:08:47.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T07:08:47.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2021-07-26T07:08:47.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T07:08:47.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:08:47.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:08:47.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:08:47.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T07:08:47.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:08:48.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:08:48.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:08:48.690 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:08:48.691 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:08:48.691 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:48.691 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:48.691 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:08:48.692 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:08:48.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:08:48.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5949: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2021-07-26T07:08:48.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:08:48.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:08:48.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:08:48.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:08:48.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:08:48.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:08:48.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:08:48.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:08:48.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:08:48.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:08:48.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:08:48.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:08:48.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:08:48.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:08:48.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:08:48.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:08:48.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:08:48.722 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:08:48.722 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:08:48.722 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:48.722 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:48.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:08:48.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:08:48.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:08:48.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:08:48.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:08:48.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:08:48.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:08:48.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:08:48.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:08:48.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:08:48.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:08:48.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:08:48.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:08:48.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:08:48.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e93456b6-bc3d-4c57-be1c-15861edb31ad 2021-07-26T07:08:48.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e93456b6-bc3d-4c57-be1c-15861edb31ad' 2021-07-26T07:08:48.727 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 e93456b6-bc3d-4c57-be1c-15861edb31ad 2021-07-26T07:08:48.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:08:48.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCAX/5gIeaIKxAAT9rXl4VuaMtdnznptVCX+Q== 2021-07-26T07:08:48.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCAX/5gIeaIKxAAT9rXl4VuaMtdnznptVCX+Q=="}' 2021-07-26T07:08:48.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e93456b6-bc3d-4c57-be1c-15861edb31ad -i td/osd-scrub-repair/0/new.json 2021-07-26T07:08:49.044 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:08:49.055 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T07:08:49.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCAX/5gIeaIKxAAT9rXl4VuaMtdnznptVCX+Q== --osd-uuid e93456b6-bc3d-4c57-be1c-15861edb31ad 2021-07-26T07:08:49.081 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:49.080+0000 7fcb1eb940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:08:49.083 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:49.082+0000 7fcb1eb940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:08:49.084 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:49.083+0000 7fcb1eb940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:08:49.084 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:49.084+0000 7fcb1eb940c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:08:49.085 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:49.084+0000 7fcb1eb940c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:08:49.085 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:49.084+0000 7fcb1eb940c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:08:49.161 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:49.160+0000 7fcb1eb940c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T07:08:54.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T07:08:54.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:08:54.818 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:08:54.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:08:54.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:08:55.160 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:08:55.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:08:55.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:08:55.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:08:55.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:08:55.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:08:55.182 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:55.181+0000 7fd9c4ef50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:08:55.183 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:55.183+0000 7fd9c4ef50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:08:55.184 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:55.184+0000 7fd9c4ef50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:08:55.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:08:55.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:08:55.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:08:55.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:08:55.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:08:55.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:08:55.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:08:55.488 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:08:55.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:08:55.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:08:55.704 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:55.704+0000 7fd9c4ef50c0 -1 Falling back to public interface 2021-07-26T07:08:55.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:08:56.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:08:56.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:08:56.755 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:08:56.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:08:56.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:08:56.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:08:57.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:08:58.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:08:58.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:08:58.033 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:08:58.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:08:58.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:08:58.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:08:58.109 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:08:58.108+0000 7fd9c4ef50c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:08:58.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:08:59.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:08:59.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:08:59.352 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:08:59.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:08:59.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:08:59.353 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:08:59.645 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/376972,v1:127.0.0.1:6803/376972] [v2:127.0.0.1:6804/376972,v1:127.0.0.1:6805/376972] exists,up e93456b6-bc3d-4c57-be1c-15861edb31ad 2021-07-26T07:08:59.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:08:59.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:08:59.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:08:59.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2021-07-26T07:08:59.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:08:59.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:08:59.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:08:59.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:08:59.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:08:59.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:08:59.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:08:59.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:08:59.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:08:59.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:08:59.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:08:59.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:08:59.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:08:59.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:08:59.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:08:59.654 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:08:59.654 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:08:59.654 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:08:59.655 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:08:59.655 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:08:59.655 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:08:59.655 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:08:59.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:08:59.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:08:59.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:08:59.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:08:59.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:08:59.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:08:59.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:08:59.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:08:59.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:08:59.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:08:59.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:08:59.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:08:59.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:08:59.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c54912f7-d618-4c24-826d-112ed678eadb 2021-07-26T07:08:59.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c54912f7-d618-4c24-826d-112ed678eadb' 2021-07-26T07:08:59.659 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 c54912f7-d618-4c24-826d-112ed678eadb 2021-07-26T07:08:59.659 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:08:59.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLX/5g/0WGJxAAwlRh9OeeTZ1+eFnlyeSfTg== 2021-07-26T07:08:59.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLX/5g/0WGJxAAwlRh9OeeTZ1+eFnlyeSfTg=="}' 2021-07-26T07:08:59.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c54912f7-d618-4c24-826d-112ed678eadb -i td/osd-scrub-repair/1/new.json 2021-07-26T07:09:00.019 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:09:00.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T07:09:00.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCLX/5g/0WGJxAAwlRh9OeeTZ1+eFnlyeSfTg== --osd-uuid c54912f7-d618-4c24-826d-112ed678eadb 2021-07-26T07:09:00.056 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:00.055+0000 7f3a68e3d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:00.057 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:00.057+0000 7f3a68e3d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:00.058 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:00.057+0000 7f3a68e3d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:00.059 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:00.058+0000 7f3a68e3d0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:09:00.059 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:00.058+0000 7f3a68e3d0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:09:00.060 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:00.058+0000 7f3a68e3d0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:09:00.138 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:00.137+0000 7f3a68e3d0c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T07:09:05.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T07:09:05.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:09:05.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:09:05.592 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T07:09:05.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:09:05.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:09:05.934 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T07:09:05.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:09:05.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:09:05.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:09:05.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:09:05.955 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:05.955+0000 7f50cd6330c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:05.957 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:05.956+0000 7f50cd6330c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:05.957 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:05.957+0000 7f50cd6330c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:06.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:09:06.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:09:06.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:09:06.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:09:06.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:09:06.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:06.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:09:06.204 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:09:06.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:06.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:09:06.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:06.479 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:06.479+0000 7f50cd6330c0 -1 Falling back to public interface 2021-07-26T07:09:07.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:07.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:07.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:09:07.473 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:09:07.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:07.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:09:07.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:08.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:08.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:08.751 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:09:08.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:09:08.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:08.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:09:08.826 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:08.825+0000 7f50cd6330c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:09:09.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:10.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:10.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:10.032 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:09:10.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:09:10.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:10.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:09:10.307 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/378371,v1:127.0.0.1:6811/378371] [v2:127.0.0.1:6812/378371,v1:127.0.0.1:6813/378371] exists,up c54912f7-d618-4c24-826d-112ed678eadb 2021-07-26T07:09:10.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:09:10.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:09:10.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:09:10.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2021-07-26T07:09:10.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T07:09:10.605 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T07:09:10.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T07:09:10.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T07:09:11.040 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T07:09:11.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:09:12.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T07:09:13.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5952: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2021-07-26T07:09:13.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:09:13.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:09:13.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:09:13.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:09:13.617 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:09:13.618 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:09:13.618 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:09:13.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:09:13.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:09:13.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:09:13.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:09:13.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:09:13.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:09:13.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:09:13.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:09:14.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:09:14.023 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:09:14.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:09:14.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:09:14.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:09:14.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836484 2021-07-26T07:09:14.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836484 2021-07-26T07:09:14.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484' 2021-07-26T07:09:14.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:09:14.183 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:09:14.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672962 2021-07-26T07:09:14.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672962 2021-07-26T07:09:14.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484 1-42949672962' 2021-07-26T07:09:14.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:09:14.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836484 2021-07-26T07:09:14.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:09:14.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:09:14.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836484 2021-07-26T07:09:14.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:09:14.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836484 2021-07-26T07:09:14.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2021-07-26T07:09:14.304 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836484 2021-07-26T07:09:14.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:09:14.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836483 -lt 21474836484 2021-07-26T07:09:14.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:09:15.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:09:15.571 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:09:15.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836485 -lt 21474836484 2021-07-26T07:09:15.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:09:15.876 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672962 2021-07-26T07:09:15.876 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:09:15.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:09:15.877 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672962 2021-07-26T07:09:15.877 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:09:15.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672962 2021-07-26T07:09:15.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2021-07-26T07:09:15.879 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672962 2021-07-26T07:09:15.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:09:16.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672963 -lt 42949672962 2021-07-26T07:09:16.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:09:16.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:09:16.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:09:16.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T07:09:16.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:09:16.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:09:16.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:09:16.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:09:16.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:09:16.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:09:16.512 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:09:16.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T07:09:16.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:09:16.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:09:16.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:09:17.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T07:09:17.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:09:17.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:09:17.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2021-07-26T07:09:17.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2021-07-26T07:09:17.660 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'foo' created 2021-07-26T07:09:17.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:09:18.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2021-07-26T07:09:18.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2021-07-26T07:09:19.131 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'csr_pool' created 2021-07-26T07:09:19.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:09:20.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5956: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2021-07-26T07:09:20.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:09:20.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:09:20.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:09:20.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:09:20.144 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:09:20.145 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:09:20.145 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:09:20.145 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:09:20.145 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:09:20.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:09:20.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:09:20.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:09:20.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:09:20.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:09:20.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:09:20.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:09:20.553 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:09:20.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:09:20.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:09:20.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:09:20.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836487 2021-07-26T07:09:20.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836487 2021-07-26T07:09:20.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487' 2021-07-26T07:09:20.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:09:20.721 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:09:20.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672965 2021-07-26T07:09:20.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672965 2021-07-26T07:09:20.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2021-07-26T07:09:20.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:09:20.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836487 2021-07-26T07:09:20.841 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:09:20.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:09:20.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836487 2021-07-26T07:09:20.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:09:20.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836487 2021-07-26T07:09:20.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-07-26T07:09:20.843 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836487 2021-07-26T07:09:20.843 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:09:21.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836485 -lt 21474836487 2021-07-26T07:09:21.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:09:22.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:09:22.111 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:09:22.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836487 2021-07-26T07:09:22.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:09:22.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672965 2021-07-26T07:09:22.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:09:22.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:09:22.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672965 2021-07-26T07:09:22.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:09:22.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672965 2021-07-26T07:09:22.389 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672965 2021-07-26T07:09:22.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-07-26T07:09:22.390 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:09:22.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672965 -lt 42949672965 2021-07-26T07:09:22.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:09:22.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:09:22.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:09:23.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:09:23.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:09:23.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:09:23.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:09:23.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:09:23.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:09:23.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:09:23.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:09:23.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:09:23.323 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:09:23.323 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:09:23.323 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:09:23.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:09:23.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:09:23.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:09:23.668 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: seq 1 2 2021-07-26T07:09:23.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2021-07-26T07:09:23.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5959: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2021-07-26T07:09:23.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2021-07-26T07:09:23.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:09:23.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:09:23.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ1 2021-07-26T07:09:23.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:09:23.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:09:23.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:09:24.180 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:09:24.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:09:24.690 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:09:24.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:09:24.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:09:24.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:09:24.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5962: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2021-07-26T07:09:25.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5963: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2021-07-26T07:09:25.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2021-07-26T07:09:25.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5959: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2021-07-26T07:09:25.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2021-07-26T07:09:25.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:09:25.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2021-07-26T07:09:25.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ2 2021-07-26T07:09:25.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:09:25.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:09:25.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:09:25.602 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:09:25.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:09:26.121 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:09:26.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:09:26.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:09:26.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:09:26.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5962: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2021-07-26T07:09:26.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5963: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2021-07-26T07:09:26.538 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2021-07-26T07:09:26.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=csr_pool 2021-07-26T07:09:26.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=ROBJ0 2021-07-26T07:09:26.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map csr_pool ROBJ0 2021-07-26T07:09:26.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:09:26.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2021-07-26T07:09:26.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2021-07-26T07:09:26.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=csr_pool 2021-07-26T07:09:26.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=ROBJ0 2021-07-26T07:09:26.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map csr_pool ROBJ0 2021-07-26T07:09:26.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:09:27.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: local primary=1 2021-07-26T07:09:27.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2021-07-26T07:09:27.289 INFO:tasks.workunit.client.0.smithi071.stdout:created pool csr_pool snap snap1 2021-07-26T07:09:27.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5970: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2021-07-26T07:09:27.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5972: TEST_corrupt_snapset_scrub_rep: seq 1 2 2021-07-26T07:09:27.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5972: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2021-07-26T07:09:27.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2021-07-26T07:09:27.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5976: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2021-07-26T07:09:27.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5976: TEST_corrupt_snapset_scrub_rep: local osd=1 2021-07-26T07:09:27.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5978: TEST_corrupt_snapset_scrub_rep: case $i in 2021-07-26T07:09:27.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5980: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2021-07-26T07:09:27.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2021-07-26T07:09:27.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:09:27.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:09:27.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=1 2021-07-26T07:09:27.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:09:27.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2021-07-26T07:09:27.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:09:27.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:09:27.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=1 2021-07-26T07:09:27.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:09:27.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:09:27.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:09:27.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:09:27.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:09:27.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:09:27.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:09:27.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:09:27.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2021-07-26T07:09:27.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:09:27.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:09:27.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:09:27.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:09:27.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:09:27.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2021-07-26T07:09:29.276 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:09:29.276 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:09:29.276 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:09:29.277 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:09:29.277 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:09:29.277 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:09:29.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2021-07-26T07:09:29.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:09:29.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:09:29.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:09:29.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:09:29.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:09:29.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:09:29.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:09:29.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:09:29.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:09:29.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:09:29.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:09:29.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:09:29.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:09:29.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:09:29.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:09:29.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:09:29.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:09:29.809 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:09:29.809 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:09:29.809 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:09:29.810 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:09:29.810 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:09:29.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:09:29.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:09:29.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:09:29.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:09:29.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:09:29.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:09:29.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:09:29.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:09:29.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:09:29.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:09:29.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:09:29.813 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:09:29.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:09:29.813 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:09:29.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:09:29.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:09:29.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:09:29.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:09:29.825 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:29.824+0000 7f96eed990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:29.860 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:29.858+0000 7f96eed990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:29.861 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:29.859+0000 7f96eed990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:30.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:09:30.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:09:30.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:09:30.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:09:30.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:09:30.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:30.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:09:30.085 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:09:30.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:30.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:09:30.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:30.381 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:30.380+0000 7f96eed990c0 -1 Falling back to public interface 2021-07-26T07:09:31.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:31.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:31.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:09:31.350 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:09:31.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:31.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:09:31.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:32.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:32.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:32.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:09:32.625 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:09:32.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:32.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:09:32.831 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:32.830+0000 7f96eed990c0 -1 osd.1 30 log_to_monitors {default=true} 2021-07-26T07:09:32.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:33.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:33.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:33.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:09:33.937 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:09:33.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:33.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:09:34.259 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/380428,v1:127.0.0.1:6811/380428] [v2:127.0.0.1:6812/380428,v1:127.0.0.1:6813/380428] exists,up c54912f7-d618-4c24-826d-112ed678eadb 2021-07-26T07:09:34.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:09:34.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:09:34.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:09:34.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:09:34.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:09:34.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:09:34.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:09:34.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:09:34.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:09:34.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:09:34.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:09:34.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:09:34.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:09:34.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:09:34.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:09:34.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:09:34.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:09:34.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:09:34.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:09:34.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:09:34.626 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:09:34.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:09:34.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:09:34.627 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:09:34.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836491 2021-07-26T07:09:34.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836491 2021-07-26T07:09:34.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491' 2021-07-26T07:09:34.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:09:34.746 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:09:34.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888066 2021-07-26T07:09:34.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888066 2021-07-26T07:09:34.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-146028888066' 2021-07-26T07:09:34.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:09:34.865 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836491 2021-07-26T07:09:34.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:09:34.866 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:09:34.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836491 2021-07-26T07:09:34.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:09:34.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836491 2021-07-26T07:09:34.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-07-26T07:09:34.868 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 21474836491 2021-07-26T07:09:34.868 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:09:35.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836491 2021-07-26T07:09:35.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:09:36.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:09:36.135 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:09:36.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836491 -lt 21474836491 2021-07-26T07:09:36.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:09:36.406 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-146028888066 2021-07-26T07:09:36.406 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:09:36.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:09:36.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-146028888066 2021-07-26T07:09:36.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:09:36.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888066 2021-07-26T07:09:36.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 146028888066' 2021-07-26T07:09:36.409 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 146028888066 2021-07-26T07:09:36.409 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:09:36.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888066 -lt 146028888066 2021-07-26T07:09:36.717 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:09:36.717 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:09:36.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:09:37.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:09:37.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:09:37.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:09:37.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:09:37.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:09:37.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:09:37.042 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:09:37.042 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:09:37.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:09:37.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:09:37.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:09:37.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:09:37.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:09:37.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:09:37.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:09:37.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5972: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2021-07-26T07:09:37.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2021-07-26T07:09:37.656 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5976: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2021-07-26T07:09:37.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5976: TEST_corrupt_snapset_scrub_rep: local osd=0 2021-07-26T07:09:37.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5978: TEST_corrupt_snapset_scrub_rep: case $i in 2021-07-26T07:09:37.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2021-07-26T07:09:44.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5986: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2021-07-26T07:09:44.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:09:44.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:09:44.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:09:44.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:09:44.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2021-07-26T07:09:44.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:09:44.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:09:44.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:09:44.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:09:44.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:09:44.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:09:44.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:09:44.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:09:44.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:09:44.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:09:44.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:09:44.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2021-07-26T07:09:44.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:09:44.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:09:44.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:09:44.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:09:44.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:09:44.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2021-07-26T07:09:46.351 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:09:46.352 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:09:46.352 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:09:46.352 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:09:46.353 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:09:46.353 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 3.0_head,#3:00000000::::head#, (61) No data available 2021-07-26T07:09:46.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:09:46.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:09:46.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:09:46.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:09:46.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:09:46.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:09:46.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:09:46.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:09:46.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:09:46.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:09:46.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:09:46.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:09:46.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:09:46.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:09:46.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:09:46.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:09:46.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:09:46.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:09:46.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:09:46.874 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:09:46.875 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:09:46.875 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:09:46.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:09:46.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:09:46.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:09:46.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:09:46.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:09:46.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:09:46.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:09:46.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:09:46.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:09:46.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:09:46.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:09:46.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:09:46.879 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:09:46.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:09:46.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:09:46.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:09:46.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:09:46.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:09:46.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:09:46.896 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:46.896+0000 7f65370220c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:46.898 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:46.898+0000 7f65370220c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:46.899 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:46.899+0000 7f65370220c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:47.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:09:47.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:09:47.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:09:47.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:09:47.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:09:47.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:47.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:09:47.154 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:09:47.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:47.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:09:47.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:47.417 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:47.416+0000 7f65370220c0 -1 Falling back to public interface 2021-07-26T07:09:48.418 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:48.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:48.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:09:48.419 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:09:48.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:48.419 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:09:48.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:49.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:49.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:49.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:09:49.687 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:09:49.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:49.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:09:49.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:50.023 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:09:50.022+0000 7f65370220c0 -1 osd.0 36 log_to_monitors {default=true} 2021-07-26T07:09:50.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:50.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:50.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:09:50.983 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:09:50.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:50.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:09:51.358 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [5,36) [v2:127.0.0.1:6802/381745,v1:127.0.0.1:6803/381745] [v2:127.0.0.1:6804/381745,v1:127.0.0.1:6805/381745] exists,up e93456b6-bc3d-4c57-be1c-15861edb31ad 2021-07-26T07:09:51.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:09:51.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:09:51.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:09:51.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:09:51.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:09:51.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:09:51.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:09:51.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:09:51.361 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:09:51.361 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:09:51.361 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:09:51.361 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:09:51.362 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:09:51.362 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:09:51.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:09:51.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:09:51.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:09:51.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:09:51.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:09:51.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:09:51.722 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:09:51.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:09:51.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:09:51.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:09:51.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=171798691842 2021-07-26T07:09:51.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 171798691842 2021-07-26T07:09:51.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-171798691842' 2021-07-26T07:09:51.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:09:51.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:09:51.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888070 2021-07-26T07:09:51.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888070 2021-07-26T07:09:51.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-171798691842 1-146028888070' 2021-07-26T07:09:51.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:09:51.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-171798691842 2021-07-26T07:09:51.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:09:51.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:09:51.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-171798691842 2021-07-26T07:09:51.999 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:09:51.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=171798691842 2021-07-26T07:09:52.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2021-07-26T07:09:52.000 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 171798691842 2021-07-26T07:09:52.000 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:09:52.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 171798691842 2021-07-26T07:09:52.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:09:53.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:09:53.267 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:09:53.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 171798691842 -lt 171798691842 2021-07-26T07:09:53.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:09:53.548 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-146028888070 2021-07-26T07:09:53.548 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:09:53.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:09:53.549 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-146028888070 2021-07-26T07:09:53.549 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:09:53.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888070 2021-07-26T07:09:53.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 146028888070' 2021-07-26T07:09:53.550 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 146028888070 2021-07-26T07:09:53.551 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:09:53.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888071 -lt 146028888070 2021-07-26T07:09:53.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:09:53.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:09:53.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:09:54.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:09:54.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:09:54.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:09:54.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:09:54.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:09:54.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:09:54.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:09:54.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:09:54.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:09:54.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:09:54.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:09:54.450 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:09:54.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:09:54.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:09:54.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:09:54.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2021-07-26T07:09:54.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5993: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2021-07-26T07:09:54.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: pg_scrub: local pgid=3.0 2021-07-26T07:09:54.775 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: pg_scrub: get_last_scrub_stamp 3.0 2021-07-26T07:09:54.775 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:09:54.775 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:09:54.776 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:09:54.776 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:09:55.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: pg_scrub: local last_scrub=2021-07-26T07:09:18.994426+0000 2021-07-26T07:09:55.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: pg_scrub: ceph pg scrub 3.0 2021-07-26T07:09:55.293 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 3.0 on osd.1 to scrub 2021-07-26T07:09:55.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: pg_scrub: wait_for_scrub 3.0 2021-07-26T07:09:18.994426+0000 2021-07-26T07:09:55.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=3.0 2021-07-26T07:09:55.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:09:18.994426+0000 2021-07-26T07:09:55.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:09:55.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:09:55.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:09:55.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:09:55.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:09:55.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:09:55.306 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:09:55.306 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:09:55.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:09:18.994426+0000 '>' 2021-07-26T07:09:18.994426+0000 2021-07-26T07:09:55.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:09:56.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:09:56.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:09:56.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:09:56.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:09:56.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:09:56.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:09:56.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:09:56.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:09:18.994426+0000 '>' 2021-07-26T07:09:18.994426+0000 2021-07-26T07:09:56.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:09:57.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:09:57.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:09:57.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:09:57.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:09:57.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:09:57.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:09:57.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:09:58.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:09:18.994426+0000 '>' 2021-07-26T07:09:18.994426+0000 2021-07-26T07:09:58.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:09:59.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:09:59.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:09:59.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2021-07-26T07:09:59.159 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=3.0 2021-07-26T07:09:59.159 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:09:59.159 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:09:59.159 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2021-07-26T07:09:59.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:09:57.792267+0000 '>' 2021-07-26T07:09:18.994426+0000 2021-07-26T07:09:59.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:09:59.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5995: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2021-07-26T07:09:59.454 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5997: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2021-07-26T07:09:59.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5997: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2021-07-26T07:09:59.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5999: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2021-07-26T07:09:59.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5999: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2021-07-26T07:09:59.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6001: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2021-07-26T07:09:59.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6003: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2021-07-26T07:09:59.498 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T07:09:59.498 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T07:09:59.498 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T07:09:59.498 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T07:09:59.499 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T07:09:59.499 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T07:09:59.499 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T07:09:59.499 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T07:09:59.500 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T07:09:59.500 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T07:09:59.500 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T07:09:59.500 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2021-07-26T07:09:59.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6003: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2021-07-26T07:09:59.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6003: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T07:09:59.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2021-07-26T07:09:59.522 INFO:tasks.workunit.client.0.smithi071.stderr: . as $in 2021-07-26T07:09:59.523 INFO:tasks.workunit.client.0.smithi071.stderr: | if type == "object" then 2021-07-26T07:09:59.523 INFO:tasks.workunit.client.0.smithi071.stderr: reduce keys[] as $key 2021-07-26T07:09:59.523 INFO:tasks.workunit.client.0.smithi071.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2021-07-26T07:09:59.523 INFO:tasks.workunit.client.0.smithi071.stderr: elif type == "array" then map( walk(f) ) | f 2021-07-26T07:09:59.524 INFO:tasks.workunit.client.0.smithi071.stderr: else f 2021-07-26T07:09:59.524 INFO:tasks.workunit.client.0.smithi071.stderr: end; 2021-07-26T07:09:59.524 INFO:tasks.workunit.client.0.smithi071.stderr:walk(if type == "object" then del(.mtime) else . end) 2021-07-26T07:09:59.525 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2021-07-26T07:09:59.525 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2021-07-26T07:09:59.525 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.version) else . end) 2021-07-26T07:09:59.525 INFO:tasks.workunit.client.0.smithi071.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2021-07-26T07:09:59.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2021-07-26T07:09:59.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2021-07-26T07:09:59.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6170: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T07:09:59.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2021-07-26T07:09:59.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6171: TEST_corrupt_snapset_scrub_rep: test no = yes 2021-07-26T07:09:59.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: test '' = yes 2021-07-26T07:09:59.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2021-07-26T07:09:59.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6182: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2021-07-26T07:09:59.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2021-07-26T07:09:59.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6184: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2021-07-26T07:09:59.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6185: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2021-07-26T07:09:59.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6186: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2021-07-26T07:09:59.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6187: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2021-07-26T07:09:59.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6188: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2021-07-26T07:09:59.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6190: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2021-07-26T07:09:59.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6192: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2021-07-26T07:09:59.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6190: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2021-07-26T07:09:59.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6192: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2021-07-26T07:09:59.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6190: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2021-07-26T07:09:59.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6192: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2021-07-26T07:09:59.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6190: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2021-07-26T07:09:59.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6192: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2021-07-26T07:09:59.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6190: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2021-07-26T07:09:59.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6192: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2021-07-26T07:09:59.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6190: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2021-07-26T07:09:59.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6192: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2021-07-26T07:09:59.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6199: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2021-07-26T07:09:59.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6205: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2021-07-26T07:09:59.971 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'csr_pool' removed 2021-07-26T07:09:59.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T07:09:59.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:09:59.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:09:59.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:09:59.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:09:59.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:09:59.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:09:59.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:09:59.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:10:00.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:10:00.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:10:00.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:10:00.090 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:10:00.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:10:00.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:10:00.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:10:00.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:10:00.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:10:00.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:10:00.095 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:10:00.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:10:00.095 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:10:00.096 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:10:00.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:10:00.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:00.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:00.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:10:00.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:10:00.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:10:00.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T07:10:00.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T07:10:00.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T07:10:00.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T07:10:00.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:10:00.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:10:00.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:10:00.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:10:00.107 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:10:00.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:10:00.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:10:00.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:10:00.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:10:00.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:10:00.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:10:00.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:10:00.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:10:00.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:10:00.111 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:10:00.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:10:00.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:10:00.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:10:00.112 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:10:00.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:10:00.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:10:00.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:10:00.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:10:00.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:00.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:00.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:10:00.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:10:00.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:10:00.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T07:10:00.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:10:00.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:00.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:00.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T07:10:00.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:10:00.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:10:00.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:10:00.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T07:10:00.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2021-07-26T07:10:00.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2021-07-26T07:10:00.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2021-07-26T07:10:00.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: local allow_overwrites=false 2021-07-26T07:10:00.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: list_missing_erasure_coded: local poolname=ecpool 2021-07-26T07:10:00.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:929: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2021-07-26T07:10:00.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T07:10:00.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:10:00.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:10:00.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:10:00.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T07:10:00.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T07:10:00.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:10:00.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:10:00.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:10:00.469 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:10:00.469 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:00.469 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:00.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:00.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:10:00.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:10:00.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:10:00.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:10:00.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:10:00.730 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:10:00.730 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:10:00.730 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:10:00.730 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:10:00.731 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:10:00.731 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:10:00.731 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:00.731 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:00.732 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:10:00.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:10:00.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T07:10:00.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:10:00.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:10:00.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:10:00.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:10:00.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:10:00.828 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:10:00.828 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:10:00.829 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:10:00.829 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:10:00.829 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:00.829 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:00.830 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:10:00.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:10:00.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T07:10:00.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:930: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2021-07-26T07:10:00.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T07:10:00.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:10:00.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:10:00.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:10:00.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T07:10:00.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:10:01.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:10:01.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:10:01.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:10:01.758 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:10:01.759 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:01.759 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:01.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:01.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:10:01.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:10:01.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: seq 0 2 2021-07-26T07:10:01.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:10:01.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: '[' false = true ']' 2021-07-26T07:10:01.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: run_osd_filestore td/osd-scrub-repair 0 2021-07-26T07:10:01.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T07:10:01.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T07:10:01.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2021-07-26T07:10:01.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T07:10:01.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:10:01.788 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:10:01.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:10:01.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:10:01.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:10:01.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:10:01.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:10:01.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:10:01.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T07:10:01.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T07:10:01.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:10:01.791 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T07:10:01.791 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:10:01.791 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:10:01.791 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:10:01.791 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:01.792 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:01.792 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:01.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:01.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T07:10:01.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T07:10:01.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T07:10:01.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:10:01.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:10:01.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:10:01.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:10:01.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:10:01.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T07:10:01.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T07:10:01.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:10:01.796 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T07:10:01.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=111ecd6a-c178-4ccd-b98f-e731ed5a4b90 2021-07-26T07:10:01.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd1 111ecd6a-c178-4ccd-b98f-e731ed5a4b90' 2021-07-26T07:10:01.797 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 111ecd6a-c178-4ccd-b98f-e731ed5a4b90 2021-07-26T07:10:01.798 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T07:10:01.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQDJX/5gsjO3LxAA+v4SKdwLLU/zppnSYLKY5g== 2021-07-26T07:10:01.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQDJX/5gsjO3LxAA+v4SKdwLLU/zppnSYLKY5g=="}' 2021-07-26T07:10:01.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 111ecd6a-c178-4ccd-b98f-e731ed5a4b90 -i td/osd-scrub-repair/0/new.json 2021-07-26T07:10:02.053 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:10:02.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2021-07-26T07:10:02.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDJX/5gsjO3LxAA+v4SKdwLLU/zppnSYLKY5g== --osd-uuid 111ecd6a-c178-4ccd-b98f-e731ed5a4b90 --osd-objectstore=filestore 2021-07-26T07:10:02.089 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:02.088+0000 7f4810b160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:02.090 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:02.090+0000 7f4810b160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:02.091 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:02.091+0000 7f4810b160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:02.572 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:02.571+0000 7f4810b160c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:10:03.125 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:03.125+0000 7f4810b160c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:10:03.126 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:03.125+0000 7f4810b160c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:10:03.126 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:03.125+0000 7f4810b160c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:10:03.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T07:10:03.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T07:10:03.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd0 key to auth repository 2021-07-26T07:10:03.642 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:10:03.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:10:03.816 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2021-07-26T07:10:03.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.0 2021-07-26T07:10:03.818 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:10:03.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T07:10:03.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:10:03.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T07:10:03.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T07:10:03.841 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:03.840+0000 7efec5c320c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:03.843 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:03.842+0000 7efec5c320c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:03.844 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:03.843+0000 7efec5c320c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:03.845 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:03.844+0000 7efec5c320c0 -1 Falling back to public interface 2021-07-26T07:10:04.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 0 2021-07-26T07:10:04.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:10:04.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:10:04.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:10:04.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:10:04.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:10:04.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:10:04.158 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:10:04.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:10:04.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:10:04.340 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:04.339+0000 7efec5c320c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:10:04.340 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:04.340+0000 7efec5c320c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:10:04.341 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:04.340+0000 7efec5c320c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:10:04.348 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:04.348+0000 7efec5c320c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:10:04.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:10:05.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:10:05.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:10:05.410 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:10:05.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:10:05.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:10:05.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:10:05.594 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/383210,v1:127.0.0.1:6801/383210] [v2:127.0.0.1:6802/383210,v1:127.0.0.1:6803/383210] exists,up 111ecd6a-c178-4ccd-b98f-e731ed5a4b90 2021-07-26T07:10:05.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:10:05.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:10:05.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:10:05.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:10:05.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: '[' false = true ']' 2021-07-26T07:10:05.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: run_osd_filestore td/osd-scrub-repair 1 2021-07-26T07:10:05.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T07:10:05.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T07:10:05.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2021-07-26T07:10:05.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T07:10:05.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:10:05.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:10:05.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:10:05.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:10:05.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:10:05.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:10:05.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:10:05.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:10:05.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T07:10:05.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T07:10:05.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:10:05.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T07:10:05.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:10:05.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:10:05.604 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:10:05.605 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:05.605 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:05.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:05.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:05.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T07:10:05.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T07:10:05.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T07:10:05.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:10:05.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:10:05.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:10:05.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:10:05.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:10:05.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T07:10:05.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T07:10:05.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:10:05.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T07:10:05.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=73e9ff0f-9a91-42dd-b175-c07d65e32ebb 2021-07-26T07:10:05.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd1 73e9ff0f-9a91-42dd-b175-c07d65e32ebb' 2021-07-26T07:10:05.609 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 73e9ff0f-9a91-42dd-b175-c07d65e32ebb 2021-07-26T07:10:05.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T07:10:05.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQDNX/5gyQGBJBAAUj5Cw3JTC/vCUIQgjoswXA== 2021-07-26T07:10:05.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQDNX/5gyQGBJBAAUj5Cw3JTC/vCUIQgjoswXA=="}' 2021-07-26T07:10:05.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 73e9ff0f-9a91-42dd-b175-c07d65e32ebb -i td/osd-scrub-repair/1/new.json 2021-07-26T07:10:05.871 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:10:05.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2021-07-26T07:10:05.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDNX/5gyQGBJBAAUj5Cw3JTC/vCUIQgjoswXA== --osd-uuid 73e9ff0f-9a91-42dd-b175-c07d65e32ebb --osd-objectstore=filestore 2021-07-26T07:10:05.909 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:05.909+0000 7fb78a8f70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:05.921 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:05.921+0000 7fb78a8f70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:05.922 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:05.922+0000 7fb78a8f70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:06.340 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:06.340+0000 7fb78a8f70c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:10:07.149 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:07.148+0000 7fb78a8f70c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:10:07.154 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:07.153+0000 7fb78a8f70c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:10:07.154 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:07.153+0000 7fb78a8f70c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:10:07.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T07:10:07.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T07:10:07.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd1 key to auth repository 2021-07-26T07:10:07.829 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T07:10:07.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:10:08.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.1 2021-07-26T07:10:08.170 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T07:10:08.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T07:10:08.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:10:08.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T07:10:08.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T07:10:08.191 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:08.190+0000 7fc6de51b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:08.193 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:08.192+0000 7fc6de51b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:08.194 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:08.193+0000 7fc6de51b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:08.195 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:08.194+0000 7fc6de51b0c0 -1 Falling back to public interface 2021-07-26T07:10:08.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 1 2021-07-26T07:10:08.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:10:08.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:10:08.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:10:08.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:10:08.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:10:08.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:10:08.575 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:10:08.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:10:08.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:10:08.771 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:08.769+0000 7fc6de51b0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:10:08.780 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:08.778+0000 7fc6de51b0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:10:08.780 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:08.778+0000 7fc6de51b0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:10:08.785 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:08.784+0000 7fc6de51b0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:10:08.858 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:10:09.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:10:09.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:10:09.860 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:10:09.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:10:09.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:10:09.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:10:10.135 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/383581,v1:127.0.0.1:6811/383581] [v2:127.0.0.1:6812/383581,v1:127.0.0.1:6813/383581] exists,up 73e9ff0f-9a91-42dd-b175-c07d65e32ebb 2021-07-26T07:10:10.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:10:10.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:10:10.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:10:10.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:10:10.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: '[' false = true ']' 2021-07-26T07:10:10.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: run_osd_filestore td/osd-scrub-repair 2 2021-07-26T07:10:10.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T07:10:10.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T07:10:10.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2021-07-26T07:10:10.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T07:10:10.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:10:10.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:10:10.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:10:10.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:10:10.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:10:10.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T07:10:10.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T07:10:10.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:10:10.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T07:10:10.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T07:10:10.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:10:10.145 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T07:10:10.145 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:10:10.145 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:10:10.145 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:10:10.145 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:10.146 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:10.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:10.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:10.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T07:10:10.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T07:10:10.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T07:10:10.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:10:10.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:10:10.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:10:10.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:10:10.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:10:10.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T07:10:10.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T07:10:10.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2021-07-26T07:10:10.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T07:10:10.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=13aeadf8-ec36-4aa8-9198-eb350f2883d7 2021-07-26T07:10:10.150 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 13aeadf8-ec36-4aa8-9198-eb350f2883d7 2021-07-26T07:10:10.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd1 13aeadf8-ec36-4aa8-9198-eb350f2883d7' 2021-07-26T07:10:10.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T07:10:10.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQDSX/5gC3EgCRAAXTEYzB+MjtXHyCD+DXC9iQ== 2021-07-26T07:10:10.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQDSX/5gC3EgCRAAXTEYzB+MjtXHyCD+DXC9iQ=="}' 2021-07-26T07:10:10.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 13aeadf8-ec36-4aa8-9198-eb350f2883d7 -i td/osd-scrub-repair/2/new.json 2021-07-26T07:10:10.505 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:10:10.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2021-07-26T07:10:10.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDSX/5gC3EgCRAAXTEYzB+MjtXHyCD+DXC9iQ== --osd-uuid 13aeadf8-ec36-4aa8-9198-eb350f2883d7 --osd-objectstore=filestore 2021-07-26T07:10:10.542 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:10.540+0000 7fb4937940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:10.554 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:10.553+0000 7fb4937940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:10.555 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:10.554+0000 7fb4937940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:11.293 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:11.291+0000 7fb4937940c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:10:11.779 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:11.777+0000 7fb4937940c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:10:11.779 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:11.777+0000 7fb4937940c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:10:11.779 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:11.777+0000 7fb4937940c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:10:12.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T07:10:12.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T07:10:12.342 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T07:10:12.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd2 key to auth repository 2021-07-26T07:10:12.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:10:12.672 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2021-07-26T07:10:12.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.2 2021-07-26T07:10:12.674 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T07:10:12.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:10:12.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T07:10:12.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T07:10:12.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T07:10:12.696 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:12.694+0000 7f5a406350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:12.697 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:12.696+0000 7f5a406350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:12.698 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:12.697+0000 7f5a406350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:12.699 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:12.698+0000 7f5a406350c0 -1 Falling back to public interface 2021-07-26T07:10:13.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 2 2021-07-26T07:10:13.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:10:13.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:10:13.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:10:13.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:10:13.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:10:13.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:10:13.092 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:10:13.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:10:13.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:10:13.244 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:13.243+0000 7f5a406350c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:10:13.252 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:13.251+0000 7f5a406350c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:10:13.253 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:13.251+0000 7f5a406350c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:10:13.258 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:13.257+0000 7f5a406350c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T07:10:13.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:10:14.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:10:14.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:10:14.401 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:10:14.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:10:14.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:10:14.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:10:14.698 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/383864,v1:127.0.0.1:6819/383864] [v2:127.0.0.1:6820/383864,v1:127.0.0.1:6821/383864] exists,up 13aeadf8-ec36-4aa8-9198-eb350f2883d7 2021-07-26T07:10:14.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:10:14.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:10:14.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:10:14.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:938: list_missing_erasure_coded: create_rbd_pool 2021-07-26T07:10:14.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T07:10:15.005 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T07:10:15.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T07:10:15.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T07:10:15.451 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T07:10:15.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:10:16.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T07:10:17.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: wait_for_clean 2021-07-26T07:10:17.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:10:17.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:10:17.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:10:17.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:10:17.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:10:17.989 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:10:17.989 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:10:17.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:10:17.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:10:17.989 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:10:18.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:10:18.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:10:18.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:10:18.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:10:18.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:10:18.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:10:18.365 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:10:18.365 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:10:18.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:10:18.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:18.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:10:18.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836484 2021-07-26T07:10:18.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836484 2021-07-26T07:10:18.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484' 2021-07-26T07:10:18.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:18.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:10:18.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672964 2021-07-26T07:10:18.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672964 2021-07-26T07:10:18.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484 1-42949672964' 2021-07-26T07:10:18.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:18.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:10:18.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509442 2021-07-26T07:10:18.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509442 2021-07-26T07:10:18.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484 1-42949672964 2-64424509442' 2021-07-26T07:10:18.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:18.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836484 2021-07-26T07:10:18.741 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:18.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:10:18.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836484 2021-07-26T07:10:18.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:18.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836484 2021-07-26T07:10:18.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2021-07-26T07:10:18.744 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836484 2021-07-26T07:10:18.745 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:10:19.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836483 -lt 21474836484 2021-07-26T07:10:19.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:10:20.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:10:20.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:10:20.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836483 -lt 21474836484 2021-07-26T07:10:20.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:10:21.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:10:21.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:10:21.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836484 -lt 21474836484 2021-07-26T07:10:21.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:21.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672964 2021-07-26T07:10:21.595 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:21.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:10:21.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672964 2021-07-26T07:10:21.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:21.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672964 2021-07-26T07:10:21.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2021-07-26T07:10:21.597 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672964 2021-07-26T07:10:21.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:10:21.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672965 -lt 42949672964 2021-07-26T07:10:21.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:21.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509442 2021-07-26T07:10:21.910 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:21.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:10:21.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509442 2021-07-26T07:10:21.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:21.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509442 2021-07-26T07:10:21.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2021-07-26T07:10:21.913 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509442 2021-07-26T07:10:21.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:10:22.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509443 -lt 64424509442 2021-07-26T07:10:22.218 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:10:22.218 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:10:22.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:10:22.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T07:10:22.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:10:22.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:10:22.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:10:22.547 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:10:22.547 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:10:22.547 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:10:22.547 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:10:22.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T07:10:22.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:10:22.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:10:22.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:10:23.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T07:10:23.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:10:23.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:10:23.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:941: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2021-07-26T07:10:23.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=ecpool 2021-07-26T07:10:23.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T07:10:23.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=false 2021-07-26T07:10:23.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T07:10:23.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2021-07-26T07:10:23.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2021-07-26T07:10:23.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2021-07-26T07:10:24.628 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' created 2021-07-26T07:10:24.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:10:25.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' false = true ']' 2021-07-26T07:10:25.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T07:10:25.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:10:25.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:10:25.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:10:25.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:10:25.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:10:25.643 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:10:25.643 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:10:25.643 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:10:25.644 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:10:25.644 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:10:25.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:10:25.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:10:25.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:10:25.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:10:25.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:10:26.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:10:26.038 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:10:26.038 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:10:26.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:10:26.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:26.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:10:26.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836486 2021-07-26T07:10:26.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836486 2021-07-26T07:10:26.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836486' 2021-07-26T07:10:26.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:26.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:10:26.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672967 2021-07-26T07:10:26.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672967 2021-07-26T07:10:26.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836486 1-42949672967' 2021-07-26T07:10:26.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:26.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:10:26.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509445 2021-07-26T07:10:26.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509445 2021-07-26T07:10:26.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836486 1-42949672967 2-64424509445' 2021-07-26T07:10:26.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:26.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836486 2021-07-26T07:10:26.404 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:26.405 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:10:26.406 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836486 2021-07-26T07:10:26.406 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:26.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836486 2021-07-26T07:10:26.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2021-07-26T07:10:26.407 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836486 2021-07-26T07:10:26.408 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:10:26.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836486 2021-07-26T07:10:26.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:26.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672967 2021-07-26T07:10:26.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:26.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:10:26.695 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672967 2021-07-26T07:10:26.696 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:26.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672967 2021-07-26T07:10:26.697 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2021-07-26T07:10:26.697 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672967 2021-07-26T07:10:26.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:10:27.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672967 -lt 42949672967 2021-07-26T07:10:27.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:27.005 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509445 2021-07-26T07:10:27.005 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:27.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:10:27.006 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509445 2021-07-26T07:10:27.007 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:27.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509445 2021-07-26T07:10:27.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2021-07-26T07:10:27.008 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509445 2021-07-26T07:10:27.008 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:10:27.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509445 -lt 64424509445 2021-07-26T07:10:27.275 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:10:27.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:10:27.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:10:27.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T07:10:27.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:10:27.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:10:27.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:10:27.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:10:27.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:10:27.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:10:27.643 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:10:27.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T07:10:27.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:10:27.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:10:27.945 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:10:28.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T07:10:28.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:10:28.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:10:28.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T07:10:28.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2021-07-26T07:10:28.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:10:28.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T07:10:28.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=MOBJ0 2021-07-26T07:10:28.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:10:28.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:10:28.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:10:29.018 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:10:29.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:10:29.521 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:10:29.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:10:29.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:10:29.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:10:29.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:945: list_missing_erasure_coded: osds0=($(get_osds $poolname MOBJ0)) 2021-07-26T07:10:29.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:945: list_missing_erasure_coded: get_osds ecpool MOBJ0 2021-07-26T07:10:29.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=ecpool 2021-07-26T07:10:29.780 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=MOBJ0 2021-07-26T07:10:29.780 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map ecpool MOBJ0 2021-07-26T07:10:29.781 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:10:30.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=2 2021-07-26T07:10:30.140 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:10:30.140 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T07:10:30.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 2 1 0 2021-07-26T07:10:30.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:945: list_missing_erasure_coded: local -a osds0 2021-07-26T07:10:30.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:948: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2021-07-26T07:10:30.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:10:30.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T07:10:30.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=MOBJ1 2021-07-26T07:10:30.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:10:30.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:10:30.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:10:30.658 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:10:30.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:10:31.194 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:10:31.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:10:31.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:10:31.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:10:31.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:949: list_missing_erasure_coded: osds1=($(get_osds $poolname MOBJ1)) 2021-07-26T07:10:31.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:949: list_missing_erasure_coded: get_osds ecpool MOBJ1 2021-07-26T07:10:31.345 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=ecpool 2021-07-26T07:10:31.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=MOBJ1 2021-07-26T07:10:31.346 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map ecpool MOBJ1 2021-07-26T07:10:31.346 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:10:31.616 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=2 2021-07-26T07:10:31.616 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:10:31.616 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T07:10:31.617 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 2 1 0 2021-07-26T07:10:31.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:949: list_missing_erasure_coded: local -a osds1 2021-07-26T07:10:31.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: list_missing_erasure_coded: seq 0 2 2021-07-26T07:10:31.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:10:31.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:10:31.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:10:31.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:10:31.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:10:31.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:10:31.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:10:31.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:10:31.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:10:31.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:10:31.725 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:10:31.725 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:10:31.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:10:31.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:10:31.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:10:31.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:10:31.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:10:31.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2021-07-26T07:10:31.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:10:31.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:10:31.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:10:31.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:10:31.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:10:31.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:10:31.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: list_missing_erasure_coded: id=2 2021-07-26T07:10:31.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2021-07-26T07:10:32.632 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:10:32.633 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:10:32.633 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:10:32.633 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:10:32.633 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T07:10:32.634 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#2:a9030505:::MOBJ0:head# 2021-07-26T07:10:32.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:959: list_missing_erasure_coded: id=1 2021-07-26T07:10:32.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2021-07-26T07:10:33.306 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:10:33.307 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T07:10:33.309 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:10:33.309 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:10:33.309 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:10:33.309 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#2:a9030505:::MOBJ0:head# 2021-07-26T07:10:33.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:963: list_missing_erasure_coded: id=1 2021-07-26T07:10:33.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2021-07-26T07:10:33.758 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:10:33.759 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T07:10:33.759 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:10:33.759 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:10:33.759 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:10:33.760 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2021-07-26T07:10:33.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:966: list_missing_erasure_coded: id=0 2021-07-26T07:10:33.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:967: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2021-07-26T07:10:34.243 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:10:34.244 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:10:34.244 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:10:34.244 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:10:34.244 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T07:10:34.245 INFO:tasks.workunit.client.0.smithi071.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2021-07-26T07:10:34.417 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 2 2021-07-26T07:10:34.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:10:34.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2021-07-26T07:10:34.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:10:34.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:10:34.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:10:34.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:10:34.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:10:34.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:10:34.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:10:34.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:10:34.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:10:34.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:10:34.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:10:34.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:10:34.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:10:34.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:10:34.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:10:34.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:10:34.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:10:34.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:10:34.427 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:10:34.427 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:34.427 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:34.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:34.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:34.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:10:34.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:10:34.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:10:34.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:10:34.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:10:34.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:10:34.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:10:34.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:10:34.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:10:34.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:10:34.430 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:10:34.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:10:34.431 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:10:34.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:10:34.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:10:34.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:10:34.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:10:34.446 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:34.446+0000 7f8dfe5a70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:34.448 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:34.448+0000 7f8dfe5a70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:34.450 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:34.450+0000 7f8dfe5a70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:34.451 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:34.451+0000 7f8dfe5a70c0 -1 Falling back to public interface 2021-07-26T07:10:34.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:10:34.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:10:34.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:10:34.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:10:34.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:10:34.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:10:34.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:10:34.765 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:10:34.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:10:34.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:10:34.804 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:34.804+0000 7f8dfe5a70c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:10:34.819 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:34.819+0000 7f8dfe5a70c0 -1 journal do_read_entry(376832): bad header magic 2021-07-26T07:10:34.820 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:34.819+0000 7f8dfe5a70c0 -1 journal do_read_entry(376832): bad header magic 2021-07-26T07:10:34.826 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:34.826+0000 7f8dfe5a70c0 -1 osd.0 33 log_to_monitors {default=true} 2021-07-26T07:10:35.042 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:10:36.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:10:36.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:10:36.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:10:36.044 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:10:36.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:10:36.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:10:36.321 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 37 up_thru 38 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6800/385311,v1:127.0.0.1:6801/385311] [v2:127.0.0.1:6802/385311,v1:127.0.0.1:6803/385311] exists,up 111ecd6a-c178-4ccd-b98f-e731ed5a4b90 2021-07-26T07:10:36.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:10:36.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:10:36.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:10:36.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:10:36.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2021-07-26T07:10:36.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:10:36.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:10:36.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:10:36.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:10:36.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:10:36.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:10:36.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:10:36.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:10:36.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:10:36.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:10:36.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:10:36.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:10:36.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:10:36.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:10:36.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:10:36.326 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:10:36.326 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:10:36.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:10:36.327 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:10:36.327 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:36.327 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:36.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:36.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:36.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:10:36.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:10:36.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:10:36.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:10:36.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:10:36.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:10:36.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:10:36.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:10:36.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:10:36.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:10:36.330 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:10:36.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:10:36.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:10:36.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:10:36.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:10:36.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:10:36.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:10:36.341 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:36.341+0000 7fa05c0fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:36.343 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:36.342+0000 7fa05c0fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:36.343 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:36.343+0000 7fa05c0fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:36.345 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:36.344+0000 7fa05c0fd0c0 -1 Falling back to public interface 2021-07-26T07:10:36.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:10:36.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:10:36.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:10:36.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:10:36.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:10:36.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:10:36.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:10:36.760 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:10:36.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:10:36.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:10:36.948 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:36.948+0000 7fa05c0fd0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:10:36.975 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:36.975+0000 7fa05c0fd0c0 -1 journal do_read_entry(360448): bad header magic 2021-07-26T07:10:36.975 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:36.975+0000 7fa05c0fd0c0 -1 journal do_read_entry(360448): bad header magic 2021-07-26T07:10:36.983 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:36.982+0000 7fa05c0fd0c0 -1 osd.1 33 log_to_monitors {default=true} 2021-07-26T07:10:37.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:10:38.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:10:38.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:10:38.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:10:38.005 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:10:38.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:10:38.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:10:38.301 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 35 last_clean_interval [10,33) [v2:127.0.0.1:6810/385479,v1:127.0.0.1:6811/385479] [v2:127.0.0.1:6812/385479,v1:127.0.0.1:6813/385479] exists,up 73e9ff0f-9a91-42dd-b175-c07d65e32ebb 2021-07-26T07:10:38.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:10:38.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:10:38.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:10:38.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:10:38.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2021-07-26T07:10:38.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:10:38.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:10:38.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T07:10:38.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:10:38.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:10:38.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:10:38.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:10:38.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:10:38.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:10:38.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T07:10:38.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T07:10:38.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:10:38.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:10:38.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:10:38.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:10:38.306 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:10:38.306 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:10:38.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:10:38.307 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:10:38.307 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:38.307 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:38.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:38.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:38.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:10:38.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:10:38.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:10:38.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:10:38.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:10:38.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:10:38.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:10:38.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:10:38.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T07:10:38.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T07:10:38.310 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.2 2021-07-26T07:10:38.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:10:38.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/2/whoami 2021-07-26T07:10:38.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T07:10:38.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:10:38.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:10:38.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:10:38.320 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:38.320+0000 7eff9fe9f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:38.322 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:38.322+0000 7eff9fe9f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:38.323 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:38.323+0000 7eff9fe9f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:38.324 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:38.324+0000 7eff9fe9f0c0 -1 Falling back to public interface 2021-07-26T07:10:38.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T07:10:38.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:10:38.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:10:38.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:10:38.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:10:38.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:10:38.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:10:38.765 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:10:38.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:10:38.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:10:39.002 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:39.002+0000 7eff9fe9f0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:10:39.013 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:39.013+0000 7eff9fe9f0c0 -1 journal do_read_entry(360448): bad header magic 2021-07-26T07:10:39.014 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:39.013+0000 7eff9fe9f0c0 -1 journal do_read_entry(360448): bad header magic 2021-07-26T07:10:39.020 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:39.020+0000 7eff9fe9f0c0 -1 osd.2 34 log_to_monitors {default=true} 2021-07-26T07:10:39.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:10:40.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:10:40.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:10:40.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:10:40.040 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:10:40.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:10:40.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:10:40.309 INFO:tasks.workunit.client.0.smithi071.stderr:osd.2 up in weight 1 up_from 45 up_thru 45 down_at 38 last_clean_interval [15,34) [v2:127.0.0.1:6818/385647,v1:127.0.0.1:6819/385647] [v2:127.0.0.1:6820/385647,v1:127.0.0.1:6821/385647] exists,up 13aeadf8-ec36-4aa8-9198-eb350f2883d7 2021-07-26T07:10:40.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:10:40.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:10:40.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:10:40.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: create_rbd_pool 2021-07-26T07:10:40.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T07:10:40.723 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' removed 2021-07-26T07:10:40.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T07:10:40.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T07:10:41.260 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T07:10:41.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:10:42.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T07:10:44.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:974: list_missing_erasure_coded: wait_for_clean 2021-07-26T07:10:44.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:10:44.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:10:44.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:10:44.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:10:44.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:10:44.753 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:10:44.753 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:10:44.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:10:44.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:10:44.754 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:10:44.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:10:44.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:10:44.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:10:44.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:10:44.864 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:10:45.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:10:45.128 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:10:45.128 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:10:45.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:10:45.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:45.129 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:10:45.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=158913789956 2021-07-26T07:10:45.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 158913789956 2021-07-26T07:10:45.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-158913789956' 2021-07-26T07:10:45.247 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:45.247 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:10:45.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=176093659139 2021-07-26T07:10:45.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 176093659139 2021-07-26T07:10:45.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-158913789956 1-176093659139' 2021-07-26T07:10:45.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:45.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:10:45.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528323 2021-07-26T07:10:45.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528323 2021-07-26T07:10:45.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-158913789956 1-176093659139 2-193273528323' 2021-07-26T07:10:45.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:45.483 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-158913789956 2021-07-26T07:10:45.483 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:45.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:10:45.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-158913789956 2021-07-26T07:10:45.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:45.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=158913789956 2021-07-26T07:10:45.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 158913789956' 2021-07-26T07:10:45.486 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 158913789956 2021-07-26T07:10:45.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:10:45.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 158913789954 -lt 158913789956 2021-07-26T07:10:45.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:10:46.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:10:46.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:10:47.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 158913789956 -lt 158913789956 2021-07-26T07:10:47.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:47.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-176093659139 2021-07-26T07:10:47.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:47.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:10:47.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-176093659139 2021-07-26T07:10:47.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:47.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=176093659139 2021-07-26T07:10:47.040 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 176093659139' 2021-07-26T07:10:47.041 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 176093659139 2021-07-26T07:10:47.041 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:10:47.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 176093659139 -lt 176093659139 2021-07-26T07:10:47.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:47.347 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-193273528323 2021-07-26T07:10:47.348 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:47.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:10:47.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-193273528323 2021-07-26T07:10:47.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:47.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528323 2021-07-26T07:10:47.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 193273528323' 2021-07-26T07:10:47.350 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 193273528323 2021-07-26T07:10:47.351 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:10:47.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528323 -lt 193273528323 2021-07-26T07:10:47.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:10:47.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:10:47.671 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:10:48.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T07:10:48.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:10:48.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:10:48.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:10:48.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:10:48.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:10:48.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:10:48.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:10:48.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T07:10:48.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:10:48.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:10:48.337 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:10:48.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T07:10:48.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:10:48.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:10:48.660 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:977: list_missing_erasure_coded: get_pg ecpool MOBJ0 2021-07-26T07:10:48.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T07:10:48.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=MOBJ0 2021-07-26T07:10:48.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool MOBJ0 2021-07-26T07:10:48.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:10:48.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:977: list_missing_erasure_coded: local pg=2.0 2021-07-26T07:10:48.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: repair 2.0 2021-07-26T07:10:48.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T07:10:48.973 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T07:10:48.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:10:48.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:10:48.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:10:48.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:10:49.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T07:10:24.439638+0000 2021-07-26T07:10:49.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T07:10:49.527 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.2 to repair 2021-07-26T07:10:49.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T07:10:24.439638+0000 2021-07-26T07:10:49.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T07:10:49.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:10:24.439638+0000 2021-07-26T07:10:49.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:10:49.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:10:49.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:10:49.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:10:49.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:10:49.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:10:49.539 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:10:49.540 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:10:49.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:10:24.439638+0000 '>' 2021-07-26T07:10:24.439638+0000 2021-07-26T07:10:49.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:10:50.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:10:50.798 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:10:50.799 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:10:50.800 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:10:50.800 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:10:50.800 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:10:50.800 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:10:51.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:10:24.439638+0000 '>' 2021-07-26T07:10:24.439638+0000 2021-07-26T07:10:51.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:10:52.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:10:52.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:10:52.112 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:10:52.112 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:10:52.112 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:10:52.113 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:10:52.113 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:10:52.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:10:24.439638+0000 '>' 2021-07-26T07:10:24.439638+0000 2021-07-26T07:10:52.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:10:53.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:10:53.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:10:53.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:10:53.388 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:10:53.389 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:10:53.390 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:10:53.390 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:10:53.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:10:24.439638+0000 '>' 2021-07-26T07:10:24.439638+0000 2021-07-26T07:10:53.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:10:54.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:10:54.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:10:54.662 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:10:54.663 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:10:54.663 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:10:54.663 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:10:54.663 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:10:54.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:10:52.138637+0000 '>' 2021-07-26T07:10:24.439638+0000 2021-07-26T07:10:54.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:10:54.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: list_missing_erasure_coded: seq 0 120 2021-07-26T07:10:54.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: list_missing_erasure_coded: for i in $(seq 0 120) 2021-07-26T07:10:54.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:984: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2021-07-26T07:10:54.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2021-07-26T07:10:54.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2021-07-26T07:10:54.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: wc -l 2021-07-26T07:10:55.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: matches=2 2021-07-26T07:10:55.064 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:986: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2021-07-26T07:10:55.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:986: list_missing_erasure_coded: break 2021-07-26T07:10:55.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T07:10:55.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:10:55.065 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:10:55.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:10:55.066 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:10:55.066 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:10:55.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:10:55.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:10:55.067 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:10:55.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:10:55.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:10:55.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:10:55.174 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:10:55.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:10:55.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:10:55.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:10:55.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:10:55.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:10:55.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:10:55.178 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:10:55.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:10:55.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:10:55.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:10:55.192 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:10:55.192 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:55.192 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:55.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:10:55.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:10:55.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:10:55.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T07:10:55.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T07:10:55.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T07:10:55.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T07:10:55.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:10:55.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:10:55.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:10:55.197 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:10:55.197 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:10:55.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:10:55.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:10:55.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:10:55.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:10:55.200 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:10:55.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:10:55.201 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:10:55.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:10:55.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:10:55.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:10:55.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:10:55.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:10:55.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:10:55.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:10:55.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:10:55.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:10:55.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:10:55.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:10:55.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:55.212 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:55.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:10:55.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:10:55.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:10:55.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T07:10:55.215 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:10:55.215 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:55.216 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:55.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T07:10:55.216 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:10:55.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:10:55.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:10:55.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T07:10:55.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2021-07-26T07:10:55.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2021-07-26T07:10:55.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2021-07-26T07:10:55.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2021-07-26T07:10:55.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: local allow_overwrites=true 2021-07-26T07:10:55.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: list_missing_erasure_coded: local poolname=ecpool 2021-07-26T07:10:55.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:929: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2021-07-26T07:10:55.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T07:10:55.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:10:55.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:10:55.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:10:55.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T07:10:55.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T07:10:55.569 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:10:55.569 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:10:55.570 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:10:55.570 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:10:55.570 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:55.571 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:55.571 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:55.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:10:55.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:10:55.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:10:55.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:10:55.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:10:55.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:10:55.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:10:55.823 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:10:55.823 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:10:55.823 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:10:55.824 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:10:55.824 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:55.824 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:55.824 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:10:55.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:10:55.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T07:10:55.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:10:55.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:10:55.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:10:55.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:10:55.923 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:10:55.924 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:10:55.924 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:10:55.924 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:10:55.924 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:10:55.925 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:55.925 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:55.925 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:10:55.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:10:55.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T07:10:56.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:930: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2021-07-26T07:10:56.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T07:10:56.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:10:56.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:10:56.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:10:56.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T07:10:56.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:10:56.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:10:56.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:10:56.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:10:56.840 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:10:56.841 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:56.841 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:56.841 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:56.842 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:10:56.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:10:56.865 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: seq 0 2 2021-07-26T07:10:56.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:10:56.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: '[' true = true ']' 2021-07-26T07:10:56.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:933: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2021-07-26T07:10:56.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:10:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:10:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:10:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:10:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:10:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:10:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:10:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:10:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:10:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:10:56.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:10:56.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:10:56.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:10:56.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:10:56.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:10:56.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:10:56.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:10:56.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:10:56.874 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:10:56.875 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:10:56.875 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:10:56.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:56.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:10:56.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:10:56.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:10:56.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:10:56.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:10:56.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:10:56.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:10:56.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:10:56.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:10:56.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:10:56.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:10:56.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:10:56.878 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:10:56.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dc2e3191-43c9-4c8b-9bfe-991aec5df9ed 2021-07-26T07:10:56.878 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dc2e3191-43c9-4c8b-9bfe-991aec5df9ed' 2021-07-26T07:10:56.879 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 dc2e3191-43c9-4c8b-9bfe-991aec5df9ed 2021-07-26T07:10:56.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:10:56.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAAYP5g06StNBAAV9OgXND+n3RZBm6VXOpRsA== 2021-07-26T07:10:56.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAAYP5g06StNBAAV9OgXND+n3RZBm6VXOpRsA=="}' 2021-07-26T07:10:56.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dc2e3191-43c9-4c8b-9bfe-991aec5df9ed -i td/osd-scrub-repair/0/new.json 2021-07-26T07:10:57.178 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:10:57.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T07:10:57.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAAYP5g06StNBAAV9OgXND+n3RZBm6VXOpRsA== --osd-uuid dc2e3191-43c9-4c8b-9bfe-991aec5df9ed 2021-07-26T07:10:57.214 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:57.213+0000 7fcba57310c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:57.215 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:57.215+0000 7fcba57310c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:57.216 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:57.216+0000 7fcba57310c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:10:57.217 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:57.216+0000 7fcba57310c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:10:57.217 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:57.216+0000 7fcba57310c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:10:57.217 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:57.216+0000 7fcba57310c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:10:57.295 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:10:57.295+0000 7fcba57310c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T07:11:02.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T07:11:02.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:11:02.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:11:02.963 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:11:02.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:11:03.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:11:03.303 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:11:03.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:11:03.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:03.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:11:03.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:11:03.324 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:03.324+0000 7f5445e310c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:03.326 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:03.326+0000 7f5445e310c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:03.327 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:03.326+0000 7f5445e310c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:03.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:11:03.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:11:03.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:11:03.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:11:03.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:11:03.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:03.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:11:03.576 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:11:03.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:03.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:11:03.843 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:03.843+0000 7f5445e310c0 -1 Falling back to public interface 2021-07-26T07:11:03.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:04.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:04.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:04.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:11:04.886 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:11:04.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:04.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:11:05.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:06.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:06.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:06.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:11:06.161 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:11:06.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:06.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:11:06.202 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:06.200+0000 7f5445e310c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:11:06.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:07.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:07.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:07.728 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:11:07.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:11:07.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:07.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:11:07.744 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/387621,v1:127.0.0.1:6803/387621] [v2:127.0.0.1:6804/387621,v1:127.0.0.1:6805/387621] exists,up dc2e3191-43c9-4c8b-9bfe-991aec5df9ed 2021-07-26T07:11:07.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:11:07.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:11:07.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:11:07.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:11:07.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: '[' true = true ']' 2021-07-26T07:11:07.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:933: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2021-07-26T07:11:07.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:11:07.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:11:07.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:11:07.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:11:07.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:11:07.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:11:07.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:11:07.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:11:07.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:11:07.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:11:07.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:11:07.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:11:07.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:11:07.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:11:07.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:11:07.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:11:07.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:11:07.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:11:07.757 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:11:07.757 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:07.758 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:11:07.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:11:07.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:11:07.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:11:07.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:11:07.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:11:07.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:11:07.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:11:07.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:11:07.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:11:07.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:07.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:11:07.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:11:07.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:11:07.761 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:11:07.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=525f0e2b-82e0-481b-b777-0699dfe23f34 2021-07-26T07:11:07.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 525f0e2b-82e0-481b-b777-0699dfe23f34' 2021-07-26T07:11:07.762 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 525f0e2b-82e0-481b-b777-0699dfe23f34 2021-07-26T07:11:07.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:11:07.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALYP5gXXCBLRAA2m4EBP1YmKAyQlfYEcUslw== 2021-07-26T07:11:07.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALYP5gXXCBLRAA2m4EBP1YmKAyQlfYEcUslw=="}' 2021-07-26T07:11:07.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 525f0e2b-82e0-481b-b777-0699dfe23f34 -i td/osd-scrub-repair/1/new.json 2021-07-26T07:11:08.112 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:11:08.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T07:11:08.127 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQALYP5gXXCBLRAA2m4EBP1YmKAyQlfYEcUslw== --osd-uuid 525f0e2b-82e0-481b-b777-0699dfe23f34 2021-07-26T07:11:08.149 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:08.148+0000 7f9ce88340c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:08.153 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:08.152+0000 7f9ce88340c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:08.154 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:08.153+0000 7f9ce88340c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:08.155 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:08.153+0000 7f9ce88340c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:11:08.155 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:08.153+0000 7f9ce88340c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:11:08.156 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:08.153+0000 7f9ce88340c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:11:08.234 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:08.233+0000 7f9ce88340c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T07:11:13.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T07:11:13.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:11:13.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:11:13.654 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T07:11:13.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:11:14.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:11:14.021 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T07:11:14.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:11:14.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:14.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:11:14.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:11:14.043 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:14.041+0000 7f157ce840c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:14.044 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:14.043+0000 7f157ce840c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:14.045 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:14.044+0000 7f157ce840c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:14.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:11:14.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:11:14.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:11:14.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:11:14.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:11:14.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:14.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:11:14.291 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:11:14.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:14.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:11:14.561 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:14.560+0000 7f157ce840c0 -1 Falling back to public interface 2021-07-26T07:11:14.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:15.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:15.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:15.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:11:15.599 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:11:15.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:15.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:11:15.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:16.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:16.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:16.876 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:11:16.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:11:16.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:16.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:11:16.913 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:16.911+0000 7f157ce840c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:11:17.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:18.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:18.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:18.229 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:11:18.229 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:11:18.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:18.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:11:18.501 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/389020,v1:127.0.0.1:6811/389020] [v2:127.0.0.1:6812/389020,v1:127.0.0.1:6813/389020] exists,up 525f0e2b-82e0-481b-b777-0699dfe23f34 2021-07-26T07:11:18.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:11:18.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:11:18.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:11:18.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:11:18.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: '[' true = true ']' 2021-07-26T07:11:18.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:933: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2021-07-26T07:11:18.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:11:18.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:11:18.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T07:11:18.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:11:18.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:11:18.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:11:18.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:11:18.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:11:18.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:11:18.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T07:11:18.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T07:11:18.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:11:18.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:11:18.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:11:18.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:11:18.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:11:18.514 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:11:18.514 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:11:18.514 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:11:18.515 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:18.515 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:11:18.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:11:18.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:11:18.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:11:18.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:11:18.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:11:18.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:11:18.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:11:18.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:11:18.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:11:18.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:18.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:11:18.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:11:18.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T07:11:18.519 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:11:18.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cb4a2154-e2fd-4aac-8f60-47c918b33474 2021-07-26T07:11:18.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cb4a2154-e2fd-4aac-8f60-47c918b33474' 2021-07-26T07:11:18.520 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 cb4a2154-e2fd-4aac-8f60-47c918b33474 2021-07-26T07:11:18.521 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:11:18.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAWYP5gDdbzHhAA0hU4XPLa7ARHF0YZlKv4+g== 2021-07-26T07:11:18.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAWYP5gDdbzHhAA0hU4XPLa7ARHF0YZlKv4+g=="}' 2021-07-26T07:11:18.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cb4a2154-e2fd-4aac-8f60-47c918b33474 -i td/osd-scrub-repair/2/new.json 2021-07-26T07:11:18.889 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:11:18.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2021-07-26T07:11:18.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAWYP5gDdbzHhAA0hU4XPLa7ARHF0YZlKv4+g== --osd-uuid cb4a2154-e2fd-4aac-8f60-47c918b33474 2021-07-26T07:11:18.926 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:18.924+0000 7f66831650c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:18.947 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:18.946+0000 7f66831650c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:18.948 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:18.946+0000 7f66831650c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:18.948 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:18.947+0000 7f66831650c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T07:11:18.949 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:18.947+0000 7f66831650c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T07:11:18.949 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:18.947+0000 7f66831650c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T07:11:19.024 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:19.023+0000 7f66831650c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2021-07-26T07:11:24.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T07:11:24.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:11:24.835 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T07:11:24.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T07:11:24.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:11:25.175 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T07:11:25.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T07:11:25.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:25.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:11:25.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:11:25.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:11:25.196 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:25.196+0000 7f00d3de00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:25.198 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:25.196+0000 7f00d3de00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:25.199 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:25.197+0000 7f00d3de00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:25.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T07:11:25.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:11:25.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:11:25.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:11:25.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:11:25.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:25.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:11:25.482 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:11:25.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:25.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:11:25.717 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:25.715+0000 7f00d3de00c0 -1 Falling back to public interface 2021-07-26T07:11:25.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:26.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:26.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:26.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:11:26.749 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:11:26.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:26.749 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:11:27.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:28.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:28.020 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:28.021 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:11:28.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:11:28.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:28.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:11:28.139 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:28.139+0000 7f00d3de00c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T07:11:28.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:29.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:29.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:29.348 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:11:29.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:11:29.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:29.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:11:29.628 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/390427,v1:127.0.0.1:6819/390427] [v2:127.0.0.1:6820/390427,v1:127.0.0.1:6821/390427] exists,up cb4a2154-e2fd-4aac-8f60-47c918b33474 2021-07-26T07:11:29.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:11:29.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:11:29.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:11:29.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:938: list_missing_erasure_coded: create_rbd_pool 2021-07-26T07:11:29.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T07:11:29.889 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T07:11:29.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T07:11:29.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T07:11:30.329 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T07:11:30.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:11:31.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T07:11:33.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: wait_for_clean 2021-07-26T07:11:33.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:11:33.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:11:33.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:11:33.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:11:33.005 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:11:33.005 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:11:33.005 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:11:33.005 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:11:33.006 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:11:33.006 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:11:33.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:11:33.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:11:33.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:11:33.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:11:33.114 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:11:33.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:11:33.379 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:11:33.380 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:11:33.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:11:33.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:11:33.380 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:11:33.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836488 2021-07-26T07:11:33.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836488 2021-07-26T07:11:33.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488' 2021-07-26T07:11:33.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:11:33.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:11:33.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672965 2021-07-26T07:11:33.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672965 2021-07-26T07:11:33.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2021-07-26T07:11:33.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:11:33.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:11:33.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509442 2021-07-26T07:11:33.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509442 2021-07-26T07:11:33.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488 1-42949672965 2-64424509442' 2021-07-26T07:11:33.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:11:33.736 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836488 2021-07-26T07:11:33.736 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:11:33.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:11:33.737 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836488 2021-07-26T07:11:33.738 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:11:33.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836488 2021-07-26T07:11:33.739 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836488 2021-07-26T07:11:33.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-07-26T07:11:33.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:11:34.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836488 2021-07-26T07:11:34.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:11:35.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:11:35.023 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:11:35.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836488 2021-07-26T07:11:35.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:11:36.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:11:36.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:11:36.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836488 -lt 21474836488 2021-07-26T07:11:36.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:11:36.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672965 2021-07-26T07:11:36.578 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:11:36.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:11:36.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672965 2021-07-26T07:11:36.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:11:36.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672965 2021-07-26T07:11:36.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-07-26T07:11:36.581 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672965 2021-07-26T07:11:36.581 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:11:36.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672965 -lt 42949672965 2021-07-26T07:11:36.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:11:36.888 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509442 2021-07-26T07:11:36.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:11:36.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:11:36.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509442 2021-07-26T07:11:36.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:11:36.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509442 2021-07-26T07:11:36.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2021-07-26T07:11:36.892 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509442 2021-07-26T07:11:36.892 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:11:37.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509443 -lt 64424509442 2021-07-26T07:11:37.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:11:37.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:37.159 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:37.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T07:11:37.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:37.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:37.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:37.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:37.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:37.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:37.488 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:37.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T07:11:37.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:37.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:37.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:38.075 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T07:11:38.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:11:38.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:11:38.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:941: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2021-07-26T07:11:38.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=ecpool 2021-07-26T07:11:38.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T07:11:38.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=true 2021-07-26T07:11:38.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T07:11:38.122 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2021-07-26T07:11:38.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2021-07-26T07:11:38.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2021-07-26T07:11:39.167 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' created 2021-07-26T07:11:39.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:11:40.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' true = true ']' 2021-07-26T07:11:40.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2021-07-26T07:11:40.574 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 2 allow_ec_overwrites to true 2021-07-26T07:11:40.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T07:11:40.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:11:40.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:11:40.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:11:40.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:11:40.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:11:40.587 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:11:40.587 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:11:40.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:11:40.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:11:40.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:11:40.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:11:40.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:11:40.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:11:40.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:11:40.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:11:40.951 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:11:40.951 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:11:40.951 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:11:40.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:11:40.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:11:40.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:11:41.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836490 2021-07-26T07:11:41.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836490 2021-07-26T07:11:41.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836490' 2021-07-26T07:11:41.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:11:41.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:11:41.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672967 2021-07-26T07:11:41.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672967 2021-07-26T07:11:41.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836490 1-42949672967' 2021-07-26T07:11:41.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:11:41.198 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:11:41.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509445 2021-07-26T07:11:41.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509445 2021-07-26T07:11:41.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836490 1-42949672967 2-64424509445' 2021-07-26T07:11:41.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:11:41.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836490 2021-07-26T07:11:41.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:11:41.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:11:41.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836490 2021-07-26T07:11:41.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:11:41.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836490 2021-07-26T07:11:41.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-07-26T07:11:41.317 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836490 2021-07-26T07:11:41.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:11:41.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836490 -lt 21474836490 2021-07-26T07:11:41.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:11:41.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672967 2021-07-26T07:11:41.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:11:41.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:11:41.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672967 2021-07-26T07:11:41.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:11:41.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672967 2021-07-26T07:11:41.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2021-07-26T07:11:41.588 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672967 2021-07-26T07:11:41.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:11:41.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672967 -lt 42949672967 2021-07-26T07:11:41.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:11:41.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509445 2021-07-26T07:11:41.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:11:41.862 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:11:41.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509445 2021-07-26T07:11:41.863 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:11:41.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509445 2021-07-26T07:11:41.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2021-07-26T07:11:41.864 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509445 2021-07-26T07:11:41.865 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:11:42.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509445 -lt 64424509445 2021-07-26T07:11:42.169 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:11:42.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:42.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:42.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T07:11:42.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:42.494 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:42.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:42.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:42.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:42.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:42.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:42.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T07:11:42.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:42.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:42.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:43.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T07:11:43.128 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:11:43.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:11:43.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T07:11:43.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2021-07-26T07:11:43.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:11:43.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T07:11:43.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=MOBJ0 2021-07-26T07:11:43.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:11:43.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:11:43.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:11:43.565 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:11:43.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:11:44.076 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:11:44.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:11:44.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:11:44.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:11:44.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:945: list_missing_erasure_coded: osds0=($(get_osds $poolname MOBJ0)) 2021-07-26T07:11:44.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:945: list_missing_erasure_coded: get_osds ecpool MOBJ0 2021-07-26T07:11:44.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=ecpool 2021-07-26T07:11:44.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=MOBJ0 2021-07-26T07:11:44.303 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map ecpool MOBJ0 2021-07-26T07:11:44.303 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:11:44.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=2 2021-07-26T07:11:44.573 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:11:44.573 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T07:11:44.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 2 1 0 2021-07-26T07:11:44.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:945: list_missing_erasure_coded: local -a osds0 2021-07-26T07:11:44.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:948: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2021-07-26T07:11:44.574 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:11:44.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T07:11:44.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=MOBJ1 2021-07-26T07:11:44.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:11:44.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:11:44.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:11:44.963 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:11:44.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:11:45.549 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:11:45.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:11:45.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:11:45.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:11:45.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:949: list_missing_erasure_coded: osds1=($(get_osds $poolname MOBJ1)) 2021-07-26T07:11:45.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:949: list_missing_erasure_coded: get_osds ecpool MOBJ1 2021-07-26T07:11:45.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=ecpool 2021-07-26T07:11:45.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=MOBJ1 2021-07-26T07:11:45.902 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map ecpool MOBJ1 2021-07-26T07:11:45.902 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:11:46.172 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=2 2021-07-26T07:11:46.172 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:11:46.172 INFO:tasks.workunit.client.0.smithi071.stderr:0' 2021-07-26T07:11:46.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 2 1 0 2021-07-26T07:11:46.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:949: list_missing_erasure_coded: local -a osds1 2021-07-26T07:11:46.174 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: list_missing_erasure_coded: seq 0 2 2021-07-26T07:11:46.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:11:46.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:11:46.175 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:11:46.175 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:11:46.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:11:46.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:11:46.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:11:46.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:11:46.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:11:46.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:11:46.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:11:46.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:11:46.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:11:46.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:11:46.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:11:46.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:11:46.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:11:46.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2021-07-26T07:11:46.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:11:46.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:11:46.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:11:46.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:11:46.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:11:46.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:11:46.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: list_missing_erasure_coded: id=2 2021-07-26T07:11:46.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2021-07-26T07:11:47.716 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:11:47.716 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s0_head,0#2:00000000::::head#, (61) No data available 2021-07-26T07:11:47.717 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:11:47.717 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:11:47.717 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:11:47.718 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#2:a9030505:::MOBJ0:head# 2021-07-26T07:11:48.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:959: list_missing_erasure_coded: id=1 2021-07-26T07:11:48.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2021-07-26T07:11:49.421 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:11:49.421 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T07:11:49.421 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:11:49.421 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:11:49.422 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:11:49.422 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#2:a9030505:::MOBJ0:head# 2021-07-26T07:11:49.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:963: list_missing_erasure_coded: id=1 2021-07-26T07:11:49.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2021-07-26T07:11:50.687 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:11:50.687 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s1_head,1#2:00000000::::head#, (61) No data available 2021-07-26T07:11:50.687 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:11:50.688 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:11:50.688 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:11:50.688 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2021-07-26T07:11:51.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:966: list_missing_erasure_coded: id=0 2021-07-26T07:11:51.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:967: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2021-07-26T07:11:52.503 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:11:52.503 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:11:52.503 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0s2_head,2#2:00000000::::head#, (61) No data available 2021-07-26T07:11:52.503 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:11:52.504 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:11:52.504 INFO:tasks.workunit.client.0.smithi071.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2021-07-26T07:11:53.019 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 2 2021-07-26T07:11:53.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:11:53.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2021-07-26T07:11:53.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:11:53.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:11:53.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:11:53.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:11:53.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:11:53.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:11:53.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:11:53.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:11:53.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:11:53.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:11:53.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:11:53.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:11:53.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:11:53.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:11:53.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:11:53.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:11:53.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:11:53.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:11:53.028 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:11:53.028 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:53.029 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:11:53.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:11:53.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:11:53.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:11:53.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:11:53.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:11:53.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:11:53.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:11:53.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:53.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:11:53.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:11:53.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:11:53.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:11:53.031 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:11:53.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:53.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:11:53.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:11:53.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:11:53.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:11:53.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:11:53.047 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:53.047+0000 7fbcd884e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:53.049 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:53.049+0000 7fbcd884e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:53.050 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:53.050+0000 7fbcd884e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:53.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:11:53.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:11:53.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:11:53.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:11:53.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:11:53.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:53.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:11:53.302 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:11:53.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:53.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:11:53.566 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:53.566+0000 7fbcd884e0c0 -1 Falling back to public interface 2021-07-26T07:11:53.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:54.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:54.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:54.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:11:54.611 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:11:54.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:54.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:11:54.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:55.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:55.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:55.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:11:55.888 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:11:55.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:55.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:11:56.144 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:56.143+0000 7fbcd884e0c0 -1 osd.0 34 log_to_monitors {default=true} 2021-07-26T07:11:56.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:57.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:57.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:57.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:11:57.168 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:11:57.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:57.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:11:57.498 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 38 up_thru 39 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6802/393650,v1:127.0.0.1:6803/393650] [v2:127.0.0.1:6804/393650,v1:127.0.0.1:6805/393650] exists,up dc2e3191-43c9-4c8b-9bfe-991aec5df9ed 2021-07-26T07:11:57.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:11:57.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:11:57.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:11:57.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:11:57.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2021-07-26T07:11:57.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:11:57.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:11:57.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:11:57.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:11:57.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:11:57.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:11:57.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:11:57.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:11:57.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:11:57.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:11:57.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:11:57.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:11:57.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:11:57.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:11:57.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:11:57.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:11:57.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:11:57.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:11:57.505 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:11:57.505 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:57.505 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:11:57.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:11:57.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:11:57.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:11:57.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:11:57.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:11:57.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:11:57.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:11:57.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:57.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:11:57.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:11:57.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:11:57.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:11:57.508 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.1 2021-07-26T07:11:57.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:57.509 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:11:57.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:11:57.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:11:57.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:11:57.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:11:57.519 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:57.519+0000 7fca5f7e20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:57.521 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:57.520+0000 7fca5f7e20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:57.522 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:57.521+0000 7fca5f7e20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:57.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:11:57.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:11:57.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:11:57.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:11:57.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:11:57.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:57.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:11:57.808 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:11:57.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:57.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:11:58.037 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:11:58.037+0000 7fca5f7e20c0 -1 Falling back to public interface 2021-07-26T07:11:58.106 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:59.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:59.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:59.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:11:59.108 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:11:59.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:59.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:11:59.383 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:00.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:00.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:00.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:12:00.385 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:12:00.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:00.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:12:00.446 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:00.445+0000 7fca5f7e20c0 -1 osd.1 34 log_to_monitors {default=true} 2021-07-26T07:12:00.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:01.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:01.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:01.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:12:01.672 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:12:01.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:01.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:12:01.956 INFO:tasks.workunit.client.0.smithi071.stderr:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 36 last_clean_interval [10,34) [v2:127.0.0.1:6810/394223,v1:127.0.0.1:6811/394223] [v2:127.0.0.1:6812/394223,v1:127.0.0.1:6813/394223] exists,up 525f0e2b-82e0-481b-b777-0699dfe23f34 2021-07-26T07:12:01.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:12:01.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:12:01.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:12:01.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for id in $(seq 0 2) 2021-07-26T07:12:01.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2021-07-26T07:12:01.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:12:01.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:12:01.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T07:12:01.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:12:01.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:12:01.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:12:01.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:12:01.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:12:01.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:12:01.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T07:12:01.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T07:12:01.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:12:01.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:12:01.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:12:01.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:12:01.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:12:01.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:12:01.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:12:01.962 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:12:01.963 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:12:01.963 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:12:01.963 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:12:01.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:12:01.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:12:01.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:12:01.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:12:01.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:12:01.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:12:01.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:12:01.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:12:01.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:12:01.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T07:12:01.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T07:12:01.966 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.2 2021-07-26T07:12:01.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:12:01.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/2/whoami 2021-07-26T07:12:01.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T07:12:01.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:12:01.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:12:01.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:12:01.977 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:01.975+0000 7f66d2ee70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:01.979 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:01.977+0000 7f66d2ee70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:01.980 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:01.978+0000 7f66d2ee70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:02.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T07:12:02.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:12:02.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:12:02.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:12:02.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:12:02.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:02.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:12:02.232 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:12:02.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:02.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:12:02.495 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:02.494+0000 7f66d2ee70c0 -1 Falling back to public interface 2021-07-26T07:12:02.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:03.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:03.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:03.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:12:03.517 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:12:03.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:03.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:12:03.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:04.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:04.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:04.795 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:12:04.795 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:12:04.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:04.796 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:12:04.927 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:04.925+0000 7f66d2ee70c0 -1 osd.2 34 log_to_monitors {default=true} 2021-07-26T07:12:05.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:06.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:06.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:06.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:12:06.073 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:12:06.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:06.074 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:12:06.467 INFO:tasks.workunit.client.0.smithi071.stderr:osd.2 up in weight 1 up_from 46 up_thru 46 down_at 39 last_clean_interval [15,34) [v2:127.0.0.1:6818/394796,v1:127.0.0.1:6819/394796] [v2:127.0.0.1:6820/394796,v1:127.0.0.1:6821/394796] exists,up cb4a2154-e2fd-4aac-8f60-47c918b33474 2021-07-26T07:12:06.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:12:06.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:12:06.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:12:06.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: create_rbd_pool 2021-07-26T07:12:06.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T07:12:06.995 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' removed 2021-07-26T07:12:07.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T07:12:07.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T07:12:07.512 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T07:12:07.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:12:08.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T07:12:10.431 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:974: list_missing_erasure_coded: wait_for_clean 2021-07-26T07:12:10.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:12:10.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:12:10.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:12:10.432 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:12:10.433 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:12:10.434 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:12:10.434 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:12:10.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:12:10.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:12:10.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:12:10.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:12:10.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:12:10.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:12:10.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:12:10.544 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:12:10.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:12:10.810 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:12:10.810 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:12:10.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:12:10.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:12:10.811 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:12:10.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=163208757252 2021-07-26T07:12:10.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 163208757252 2021-07-26T07:12:10.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-163208757252' 2021-07-26T07:12:10.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:12:10.929 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:12:11.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=180388626436 2021-07-26T07:12:11.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 180388626436 2021-07-26T07:12:11.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-163208757252 1-180388626436' 2021-07-26T07:12:11.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:12:11.048 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:12:11.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=197568495619 2021-07-26T07:12:11.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 197568495619 2021-07-26T07:12:11.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-163208757252 1-180388626436 2-197568495619' 2021-07-26T07:12:11.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:12:11.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-163208757252 2021-07-26T07:12:11.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:12:11.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:12:11.169 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-163208757252 2021-07-26T07:12:11.169 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:12:11.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=163208757252 2021-07-26T07:12:11.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 163208757252' 2021-07-26T07:12:11.170 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 163208757252 2021-07-26T07:12:11.171 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:12:11.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 163208757251 -lt 163208757252 2021-07-26T07:12:11.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:12:12.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:12:12.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:12:12.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 163208757253 -lt 163208757252 2021-07-26T07:12:12.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:12:12.713 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-180388626436 2021-07-26T07:12:12.713 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:12:12.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:12:12.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-180388626436 2021-07-26T07:12:12.714 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:12:12.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=180388626436 2021-07-26T07:12:12.716 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 180388626436 2021-07-26T07:12:12.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 180388626436' 2021-07-26T07:12:12.717 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:12:13.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 180388626436 -lt 180388626436 2021-07-26T07:12:13.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:12:13.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-197568495619 2021-07-26T07:12:13.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:12:13.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:12:13.023 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-197568495619 2021-07-26T07:12:13.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:12:13.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=197568495619 2021-07-26T07:12:13.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 197568495619' 2021-07-26T07:12:13.025 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 197568495619 2021-07-26T07:12:13.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:12:13.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 197568495619 -lt 197568495619 2021-07-26T07:12:13.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:12:13.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:12:13.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:12:13.656 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T07:12:13.657 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:12:13.657 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:12:13.657 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:12:13.657 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:12:13.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:12:13.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:12:13.658 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:12:13.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T07:12:13.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:12:13.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:12:13.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:12:14.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T07:12:14.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:12:14.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:12:14.272 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:977: list_missing_erasure_coded: get_pg ecpool MOBJ0 2021-07-26T07:12:14.272 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T07:12:14.273 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=MOBJ0 2021-07-26T07:12:14.273 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool MOBJ0 2021-07-26T07:12:14.273 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:12:14.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:977: list_missing_erasure_coded: local pg=2.0 2021-07-26T07:12:14.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: repair 2.0 2021-07-26T07:12:14.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=2.0 2021-07-26T07:12:14.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 2.0 2021-07-26T07:12:14.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:12:14.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:12:14.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:12:14.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:12:14.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T07:11:39.087474+0000 2021-07-26T07:12:14.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 2.0 2021-07-26T07:12:15.137 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0s0 on osd.2 to repair 2021-07-26T07:12:15.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 2.0 2021-07-26T07:11:39.087474+0000 2021-07-26T07:12:15.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T07:12:15.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:11:39.087474+0000 2021-07-26T07:12:15.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:12:15.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:12:15.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:12:15.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:12:15.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:12:15.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:12:15.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:12:15.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:12:15.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:11:39.087474+0000 '>' 2021-07-26T07:11:39.087474+0000 2021-07-26T07:12:15.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:12:16.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:12:16.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:12:16.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:12:16.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:12:16.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:12:16.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:12:16.414 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:12:16.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:11:39.087474+0000 '>' 2021-07-26T07:11:39.087474+0000 2021-07-26T07:12:16.723 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:12:17.724 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:12:17.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:12:17.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:12:17.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:12:17.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:12:17.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:12:17.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:12:17.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:11:39.087474+0000 '>' 2021-07-26T07:11:39.087474+0000 2021-07-26T07:12:17.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:12:19.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:12:19.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:12:19.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:12:19.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:12:19.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:12:19.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:12:19.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:12:19.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:11:39.087474+0000 '>' 2021-07-26T07:11:39.087474+0000 2021-07-26T07:12:19.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:12:20.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:12:20.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:12:20.274 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:12:20.275 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:12:20.275 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:12:20.275 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:12:20.275 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:12:20.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:11:39.087474+0000 '>' 2021-07-26T07:11:39.087474+0000 2021-07-26T07:12:20.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:12:21.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:12:21.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:12:21.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:12:21.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:12:21.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:12:21.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:12:21.547 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:12:21.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:11:39.087474+0000 '>' 2021-07-26T07:11:39.087474+0000 2021-07-26T07:12:21.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:12:22.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:12:22.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:12:22.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:12:22.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:12:22.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:12:22.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:12:22.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:12:23.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:11:39.087474+0000 '>' 2021-07-26T07:11:39.087474+0000 2021-07-26T07:12:23.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:12:24.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:12:24.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:12:24.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:12:24.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:12:24.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:12:24.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:12:24.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:12:24.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:11:39.087474+0000 '>' 2021-07-26T07:11:39.087474+0000 2021-07-26T07:12:24.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:12:25.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:12:25.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:12:25.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:12:25.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:12:25.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:12:25.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:12:25.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:12:25.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:12:21.011717+0000 '>' 2021-07-26T07:11:39.087474+0000 2021-07-26T07:12:25.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:12:25.637 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: list_missing_erasure_coded: seq 0 120 2021-07-26T07:12:25.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: list_missing_erasure_coded: for i in $(seq 0 120) 2021-07-26T07:12:25.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:984: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2021-07-26T07:12:25.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2021-07-26T07:12:25.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2021-07-26T07:12:25.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: wc -l 2021-07-26T07:12:25.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: matches=2 2021-07-26T07:12:25.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:986: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2021-07-26T07:12:25.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:986: list_missing_erasure_coded: break 2021-07-26T07:12:25.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T07:12:25.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:12:25.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:12:25.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:12:25.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:12:25.763 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:12:25.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:12:25.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:12:25.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:12:25.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:12:25.869 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:12:25.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:12:25.870 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:12:25.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:12:25.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:12:25.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:12:25.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:12:25.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:12:25.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:12:25.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:12:25.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:12:25.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:12:25.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:12:25.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:12:25.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:12:25.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:12:25.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:12:25.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:12:25.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:12:25.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T07:12:25.887 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T07:12:25.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T07:12:25.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T07:12:25.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:12:25.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:12:25.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:12:25.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:12:25.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:12:25.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:12:25.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:12:25.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:12:25.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:12:25.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:12:25.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:12:25.892 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:12:25.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:12:25.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:12:25.892 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:12:25.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:12:25.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:12:25.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:12:25.894 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:12:25.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:12:25.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:12:25.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:12:25.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:12:25.897 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:12:25.897 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:12:25.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:12:25.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:12:25.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:12:25.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T07:12:25.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:12:25.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:12:25.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:12:25.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T07:12:25.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:12:25.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:12:25.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:12:25.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T07:12:25.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2021-07-26T07:12:25.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5776: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2021-07-26T07:12:25.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: local poolname=psr_pool 2021-07-26T07:12:25.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: local objname=POBJ 2021-07-26T07:12:25.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5780: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2021-07-26T07:12:25.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T07:12:25.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:12:25.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:12:25.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:12:25.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T07:12:25.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2021-07-26T07:12:26.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:12:26.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:12:26.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:12:26.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:12:26.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:12:26.263 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:12:26.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:12:26.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2021-07-26T07:12:26.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:12:26.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:12:26.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:12:26.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:12:26.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:12:26.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:12:26.524 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:12:26.524 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:12:26.524 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:12:26.524 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:12:26.525 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:12:26.525 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:12:26.525 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:12:26.526 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:12:26.526 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T07:12:26.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:12:26.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:12:26.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:12:26.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:12:26.622 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:12:26.622 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:12:26.622 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:12:26.623 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:12:26.623 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:12:26.623 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:12:26.623 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:12:26.623 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:12:26.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:12:26.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T07:12:26.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2021-07-26T07:12:26.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T07:12:26.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:12:26.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:12:26.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:12:26.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T07:12:26.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:12:27.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:12:27.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:12:27.541 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:12:27.542 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:12:27.542 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:12:27.542 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:12:27.542 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:12:27.543 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:12:27.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:12:27.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2021-07-26T07:12:27.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5783: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2021-07-26T07:12:27.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2021-07-26T07:12:27.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:12:27.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:12:27.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:12:27.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:12:27.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:12:27.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:12:27.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:12:27.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:12:27.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:12:27.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:12:27.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:12:27.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:12:27.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:12:27.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:12:27.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:12:27.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:12:27.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:12:27.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:12:27.573 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:12:27.574 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:12:27.574 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:12:27.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:12:27.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:12:27.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:12:27.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:12:27.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:12:27.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:12:27.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:12:27.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:12:27.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:12:27.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:12:27.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:12:27.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2021-07-26T07:12:27.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:12:27.577 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:12:27.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e9f079a3-1c28-4eb2-84d6-8a21bed67847 2021-07-26T07:12:27.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e9f079a3-1c28-4eb2-84d6-8a21bed67847' 2021-07-26T07:12:27.579 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 e9f079a3-1c28-4eb2-84d6-8a21bed67847 2021-07-26T07:12:27.579 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:12:27.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbYP5g1lPIIhAAyeRxFppTpWkzXNxZ5O9jZQ== 2021-07-26T07:12:27.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbYP5g1lPIIhAAyeRxFppTpWkzXNxZ5O9jZQ=="}' 2021-07-26T07:12:27.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e9f079a3-1c28-4eb2-84d6-8a21bed67847 -i td/osd-scrub-repair/0/new.json 2021-07-26T07:12:27.880 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:12:27.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T07:12:27.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQBbYP5g1lPIIhAAyeRxFppTpWkzXNxZ5O9jZQ== --osd-uuid e9f079a3-1c28-4eb2-84d6-8a21bed67847 2021-07-26T07:12:27.918 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:27.917+0000 7f6ad7cfd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:27.919 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:27.919+0000 7f6ad7cfd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:27.920 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:27.920+0000 7f6ad7cfd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:27.921 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:27.920+0000 7f6ad7cfd0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:12:27.921 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:27.920+0000 7f6ad7cfd0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:12:27.921 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:27.920+0000 7f6ad7cfd0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:12:27.998 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:27.997+0000 7f6ad7cfd0c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T07:12:33.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T07:12:33.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:12:33.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:12:33.623 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:12:33.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:12:33.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:12:33.962 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:12:33.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2021-07-26T07:12:33.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:12:33.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:12:33.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:12:33.985 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:33.985+0000 7fe1097ce0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:33.987 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:33.987+0000 7fe1097ce0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:33.988 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:33.988+0000 7fe1097ce0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:34.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:12:34.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:12:34.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:12:34.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:12:34.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:12:34.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:34.591 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:12:34.591 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:12:34.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:34.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:12:34.592 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:34.503+0000 7fe1097ce0c0 -1 Falling back to public interface 2021-07-26T07:12:34.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:35.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:35.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:35.567 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:12:35.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:12:35.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:35.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:12:35.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:36.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:36.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:36.839 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:12:36.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:12:36.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:36.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:12:36.900 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:36.899+0000 7fe1097ce0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:12:37.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:38.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:38.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:38.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:12:38.149 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:12:38.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:38.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:12:38.444 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/397286,v1:127.0.0.1:6803/397286] [v2:127.0.0.1:6804/397286,v1:127.0.0.1:6805/397286] exists,up e9f079a3-1c28-4eb2-84d6-8a21bed67847 2021-07-26T07:12:38.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:12:38.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:12:38.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:12:38.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5785: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2021-07-26T07:12:38.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:12:38.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:12:38.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:12:38.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:12:38.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:12:38.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:12:38.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:12:38.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:12:38.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:12:38.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:12:38.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:12:38.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:12:38.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:12:38.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:12:38.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:12:38.454 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:12:38.454 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:12:38.454 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:12:38.454 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:12:38.455 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:12:38.455 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:12:38.455 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:12:38.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:12:38.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:12:38.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:12:38.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:12:38.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:12:38.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:12:38.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:12:38.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:12:38.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:12:38.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:12:38.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2021-07-26T07:12:38.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:12:38.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:12:38.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=73f56971-24a2-4859-9b74-f556875bf05e 2021-07-26T07:12:38.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 73f56971-24a2-4859-9b74-f556875bf05e' 2021-07-26T07:12:38.459 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 73f56971-24a2-4859-9b74-f556875bf05e 2021-07-26T07:12:38.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:12:38.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBmYP5gEP+PGxAA4HOovv87Yj7aDSSI1XmCWw== 2021-07-26T07:12:38.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBmYP5gEP+PGxAA4HOovv87Yj7aDSSI1XmCWw=="}' 2021-07-26T07:12:38.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 73f56971-24a2-4859-9b74-f556875bf05e -i td/osd-scrub-repair/1/new.json 2021-07-26T07:12:38.827 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:12:38.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T07:12:38.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQBmYP5gEP+PGxAA4HOovv87Yj7aDSSI1XmCWw== --osd-uuid 73f56971-24a2-4859-9b74-f556875bf05e 2021-07-26T07:12:38.865 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:38.865+0000 7f0ac75ef0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:38.867 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:38.866+0000 7f0ac75ef0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:38.867 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:38.867+0000 7f0ac75ef0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:38.868 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:38.868+0000 7f0ac75ef0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:12:38.869 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:38.868+0000 7f0ac75ef0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:12:38.869 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:38.868+0000 7f0ac75ef0c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:12:38.929 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:38.928+0000 7f0ac75ef0c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T07:12:44.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T07:12:44.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:12:44.323 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T07:12:44.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:12:44.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:12:44.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:12:44.667 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T07:12:44.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:12:44.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2021-07-26T07:12:44.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:12:44.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:12:44.690 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:44.690+0000 7f8f483140c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:44.710 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:44.709+0000 7f8f483140c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:44.711 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:44.710+0000 7f8f483140c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:44.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:12:44.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:12:44.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:12:44.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:12:44.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:12:44.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:44.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:12:44.938 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:12:44.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:44.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:12:45.227 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:45.226+0000 7f8f483140c0 -1 Falling back to public interface 2021-07-26T07:12:45.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:46.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:46.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:46.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:12:46.245 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:12:46.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:46.246 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:12:46.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:47.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:47.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:47.518 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:12:47.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:12:47.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:47.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:12:47.595 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:12:47.594+0000 7f8f483140c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:12:47.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:48.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:48.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:48.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:12:48.821 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:12:48.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:48.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:12:49.116 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/398685,v1:127.0.0.1:6811/398685] [v2:127.0.0.1:6812/398685,v1:127.0.0.1:6813/398685] exists,up 73f56971-24a2-4859-9b74-f556875bf05e 2021-07-26T07:12:49.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:12:49.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:12:49.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:12:49.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5786: TEST_periodic_scrub_replicated: create_rbd_pool 2021-07-26T07:12:49.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T07:12:49.375 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T07:12:49.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T07:12:49.384 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T07:12:49.811 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T07:12:49.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:12:50.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T07:12:52.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: wait_for_clean 2021-07-26T07:12:52.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:12:52.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:12:52.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:12:52.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:12:52.060 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:12:52.060 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:12:52.061 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:12:52.061 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:12:52.061 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:12:52.061 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:12:52.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:12:52.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:12:52.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:12:52.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:12:52.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:12:52.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:12:52.456 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:12:52.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:12:52.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:12:52.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:12:52.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836485 2021-07-26T07:12:52.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836485 2021-07-26T07:12:52.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836485' 2021-07-26T07:12:52.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:12:52.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:12:52.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672962 2021-07-26T07:12:52.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672962 2021-07-26T07:12:52.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836485 1-42949672962' 2021-07-26T07:12:52.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:12:52.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836485 2021-07-26T07:12:52.771 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:12:52.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:12:52.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836485 2021-07-26T07:12:52.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:12:52.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836485 2021-07-26T07:12:52.774 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836485 2021-07-26T07:12:52.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2021-07-26T07:12:52.775 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:12:53.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836484 -lt 21474836485 2021-07-26T07:12:53.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:12:54.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:12:54.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:12:54.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836485 2021-07-26T07:12:54.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:12:54.321 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672962 2021-07-26T07:12:54.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:12:54.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:12:54.323 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672962 2021-07-26T07:12:54.323 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:12:54.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672962 2021-07-26T07:12:54.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2021-07-26T07:12:54.324 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672962 2021-07-26T07:12:54.325 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:12:54.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672963 -lt 42949672962 2021-07-26T07:12:54.617 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:12:54.617 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:12:54.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:12:54.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T07:12:54.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:12:54.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:12:54.946 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:12:54.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:12:54.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:12:54.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:12:54.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:12:55.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T07:12:55.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:12:55.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:12:55.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:12:55.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T07:12:55.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:12:55.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:12:55.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2021-07-26T07:12:55.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2021-07-26T07:12:55.998 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'psr_pool' created 2021-07-26T07:12:56.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:12:57.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5790: TEST_periodic_scrub_replicated: wait_for_clean 2021-07-26T07:12:57.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:12:57.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:12:57.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:12:57.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:12:57.010 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:12:57.011 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:12:57.012 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:12:57.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:12:57.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:12:57.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:12:57.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:12:57.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:12:57.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:12:57.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:12:57.121 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:12:57.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:12:57.410 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:12:57.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:12:57.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:12:57.410 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:12:57.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836487 2021-07-26T07:12:57.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836487 2021-07-26T07:12:57.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487' 2021-07-26T07:12:57.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:12:57.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:12:57.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672964 2021-07-26T07:12:57.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672964 2021-07-26T07:12:57.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2021-07-26T07:12:57.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:12:57.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836487 2021-07-26T07:12:57.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:12:57.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:12:57.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836487 2021-07-26T07:12:57.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:12:57.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836487 2021-07-26T07:12:57.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-07-26T07:12:57.653 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836487 2021-07-26T07:12:57.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:12:57.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836487 2021-07-26T07:12:57.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:12:58.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:12:58.963 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:12:59.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836487 2021-07-26T07:12:59.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:12:59.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672964 2021-07-26T07:12:59.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:12:59.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:12:59.243 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672964 2021-07-26T07:12:59.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:12:59.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672964 2021-07-26T07:12:59.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2021-07-26T07:12:59.245 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672964 2021-07-26T07:12:59.246 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:12:59.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672964 -lt 42949672964 2021-07-26T07:12:59.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:12:59.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:12:59.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:12:59.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T07:12:59.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:12:59.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:12:59.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:12:59.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:12:59.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:12:59.876 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:12:59.877 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:13:00.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T07:13:00.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:13:00.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:13:00.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:13:00.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 5 2021-07-26T07:13:00.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 4 '!=' -1 2021-07-26T07:13:00.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T07:13:00.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=4 2021-07-26T07:13:00.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:13:00.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:13:00.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:13:00.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:13:00.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:13:00.613 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:13:00.613 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:13:00.613 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:13:00.613 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:13:00.614 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:13:00.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T07:13:00.897 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:13:00.897 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:13:00.898 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:13:01.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T07:13:01.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:13:01.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:13:01.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5792: TEST_periodic_scrub_replicated: local osd=0 2021-07-26T07:13:01.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5793: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2021-07-26T07:13:01.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:13:01.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=psr_pool 2021-07-26T07:13:01.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=POBJ 2021-07-26T07:13:01.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=scrub 2021-07-26T07:13:01.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' scrub = noscrub ']' 2021-07-26T07:13:01.225 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:80: add_something: ceph osd unset noscrub 2021-07-26T07:13:01.656 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is unset 2021-07-26T07:13:01.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset nodeep-scrub 2021-07-26T07:13:02.174 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is unset 2021-07-26T07:13:02.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:13:02.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:13:02.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2021-07-26T07:13:02.325 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2021-07-26T07:13:02.325 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=psr_pool 2021-07-26T07:13:02.326 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=POBJ 2021-07-26T07:13:02.326 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map psr_pool POBJ 2021-07-26T07:13:02.326 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:13:02.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: local primary=0 2021-07-26T07:13:02.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5795: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2021-07-26T07:13:02.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=psr_pool 2021-07-26T07:13:02.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=POBJ 2021-07-26T07:13:02.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map psr_pool POBJ 2021-07-26T07:13:02.598 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:13:02.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5795: TEST_periodic_scrub_replicated: local pg=2.0 2021-07-26T07:13:02.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5798: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2021-07-26T07:13:02.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: echo UVWXYZ 2021-07-26T07:13:02.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5801: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:13:02.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:13:02.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:13:02.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:13:02.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:13:02.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:13:02.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:13:02.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:13:02.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:13:02.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:13:02.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:13:02.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:13:02.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:13:02.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:13:02.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:13:02.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:13:02.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:13:02.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:13:02.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:13:02.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:13:02.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:13:02.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:13:02.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:13:02.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2021-07-26T07:13:04.696 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 2.0_head,#2:00000000::::head#, (61) No data available 2021-07-26T07:13:04.696 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:13:04.696 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:13:04.697 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:13:04.697 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:13:05.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2021-07-26T07:13:05.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:13:05.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:13:05.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:13:05.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:13:05.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:13:05.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:13:05.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:13:05.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:13:05.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:13:05.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:13:05.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:13:05.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:13:05.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:13:05.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:13:05.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:13:05.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:13:05.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:13:05.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:13:05.315 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:13:05.316 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:13:05.316 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:13:05.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:13:05.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:13:05.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:13:05.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:13:05.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:13:05.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:13:05.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:13:05.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:13:05.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:13:05.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2021-07-26T07:13:05.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:13:05.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:13:05.319 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:13:05.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2021-07-26T07:13:05.319 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:13:05.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:13:05.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:13:05.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:13:05.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:13:05.334 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:05.332+0000 7fb75f5040c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:05.335 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:05.334+0000 7fb75f5040c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:05.336 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:05.335+0000 7fb75f5040c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:05.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:13:05.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:13:05.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:13:05.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:13:05.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:13:05.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:13:05.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:13:05.584 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:13:05.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:13:05.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:13:05.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:13:05.856 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:05.855+0000 7fb75f5040c0 -1 Falling back to public interface 2021-07-26T07:13:06.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:13:06.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:13:06.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:13:06.849 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:13:06.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:13:06.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:13:07.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:13:08.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:13:08.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:13:08.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:13:08.117 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:13:08.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:13:08.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:13:08.317 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:08.316+0000 7fb75f5040c0 -1 osd.0 24 log_to_monitors {default=true} 2021-07-26T07:13:08.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:13:09.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:13:09.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:13:09.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:13:09.423 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:13:09.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:13:09.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:13:09.729 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 28 up_thru 28 down_at 25 last_clean_interval [5,24) [v2:127.0.0.1:6802/400570,v1:127.0.0.1:6803/400570] [v2:127.0.0.1:6804/400570,v1:127.0.0.1:6805/400570] exists,up e9f079a3-1c28-4eb2-84d6-8a21bed67847 2021-07-26T07:13:09.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:13:09.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:13:09.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:13:09.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:13:09.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:13:09.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:13:09.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:13:09.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:13:09.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:13:09.731 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:13:09.732 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:13:09.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:13:09.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:13:09.733 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:13:09.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:13:09.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:13:09.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:13:09.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:13:09.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:13:10.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:13:10.091 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:13:10.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:13:10.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:10.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:13:10.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=120259084290 2021-07-26T07:13:10.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 120259084290 2021-07-26T07:13:10.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-120259084290' 2021-07-26T07:13:10.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:10.211 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:13:10.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672968 2021-07-26T07:13:10.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672968 2021-07-26T07:13:10.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-120259084290 1-42949672968' 2021-07-26T07:13:10.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:10.348 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-120259084290 2021-07-26T07:13:10.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:10.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:13:10.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-120259084290 2021-07-26T07:13:10.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:10.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=120259084290 2021-07-26T07:13:10.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 120259084290' 2021-07-26T07:13:10.351 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 120259084290 2021-07-26T07:13:10.351 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:13:10.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 120259084290 -lt 120259084290 2021-07-26T07:13:10.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:10.665 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672968 2021-07-26T07:13:10.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:10.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:13:10.667 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672968 2021-07-26T07:13:10.667 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:10.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672968 2021-07-26T07:13:10.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-07-26T07:13:10.668 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672968 2021-07-26T07:13:10.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:13:10.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672967 -lt 42949672968 2021-07-26T07:13:10.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:13:11.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:13:11.938 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:13:12.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672967 -lt 42949672968 2021-07-26T07:13:12.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:13:13.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:13:13.217 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:13:13.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672968 -lt 42949672968 2021-07-26T07:13:13.492 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:13:13.492 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:13:13.492 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:13:13.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T07:13:13.860 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:13:13.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:13:13.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:13:13.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:13:13.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:13:13.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:13:13.861 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:13:14.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T07:13:14.154 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:13:14.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:13:14.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:13:14.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T07:13:14.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:13:14.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:13:14.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5804: TEST_periodic_scrub_replicated: set -o pipefail 2021-07-26T07:13:14.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5805: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2021-07-26T07:13:14.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5805: TEST_periodic_scrub_replicated: jq . 2021-07-26T07:13:19.738 INFO:tasks.workunit.client.0.smithi071.stderr:No scrub information available for pg 2.0 2021-07-26T07:13:19.739 INFO:tasks.workunit.client.0.smithi071.stderr:error 2: (2) No such file or directory 2021-07-26T07:13:19.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5806: TEST_periodic_scrub_replicated: set +o pipefail 2021-07-26T07:13:19.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5808: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2021-07-26T07:13:19.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: pg_deep_scrub: local pgid=2.0 2021-07-26T07:13:19.742 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2021-07-26T07:13:19.742 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:19.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:13:19.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:19.743 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2021-07-26T07:13:20.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: pg_deep_scrub: local last_scrub=2021-07-26T07:12:55.911685+0000 2021-07-26T07:13:20.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1800: pg_deep_scrub: ceph pg deep-scrub 2.0 2021-07-26T07:13:20.266 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2021-07-26T07:13:20.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: pg_deep_scrub: wait_for_scrub 2.0 2021-07-26T07:12:55.911685+0000 last_deep_scrub_stamp 2021-07-26T07:13:20.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T07:13:20.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:12:55.911685+0000 2021-07-26T07:13:20.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_deep_scrub_stamp 2021-07-26T07:13:20.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:13:20.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:13:20.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2021-07-26T07:13:20.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:20.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:13:20.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:20.279 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2021-07-26T07:13:20.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:12:55.911685+0000 '>' 2021-07-26T07:12:55.911685+0000 2021-07-26T07:13:20.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:13:21.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:13:21.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:13:21.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2021-07-26T07:13:21.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:21.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:13:21.546 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:21.547 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2021-07-26T07:13:21.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:12:55.911685+0000 '>' 2021-07-26T07:12:55.911685+0000 2021-07-26T07:13:21.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:13:22.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:13:22.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:13:22.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2021-07-26T07:13:22.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:22.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:13:22.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:22.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2021-07-26T07:13:23.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:12:55.911685+0000 '>' 2021-07-26T07:12:55.911685+0000 2021-07-26T07:13:23.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:13:24.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:13:24.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:13:24.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2021-07-26T07:13:24.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:24.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2021-07-26T07:13:24.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:24.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2021-07-26T07:13:24.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:13:20.342139+0000 '>' 2021-07-26T07:12:55.911685+0000 2021-07-26T07:13:24.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:13:24.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5811: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2021-07-26T07:13:24.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5811: TEST_periodic_scrub_replicated: jq . 2021-07-26T07:13:24.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5811: TEST_periodic_scrub_replicated: grep -q POBJ 2021-07-26T07:13:24.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5813: TEST_periodic_scrub_replicated: flush_pg_stats 2021-07-26T07:13:24.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:13:24.390 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:13:24.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:13:24.659 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:13:24.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:13:24.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:24.660 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:13:24.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=120259084294 2021-07-26T07:13:24.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 120259084294 2021-07-26T07:13:24.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-120259084294' 2021-07-26T07:13:24.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:24.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:13:24.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672972 2021-07-26T07:13:24.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672972 2021-07-26T07:13:24.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-120259084294 1-42949672972' 2021-07-26T07:13:24.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:24.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-120259084294 2021-07-26T07:13:24.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:24.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:13:24.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-120259084294 2021-07-26T07:13:24.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:24.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=120259084294 2021-07-26T07:13:24.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 120259084294' 2021-07-26T07:13:24.902 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 120259084294 2021-07-26T07:13:24.903 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:13:25.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 120259084293 -lt 120259084294 2021-07-26T07:13:25.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:13:26.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:13:26.171 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:13:26.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 120259084294 -lt 120259084294 2021-07-26T07:13:26.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:26.476 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672972 2021-07-26T07:13:26.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:26.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:13:26.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672972 2021-07-26T07:13:26.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:26.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672972 2021-07-26T07:13:26.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2021-07-26T07:13:26.480 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672972 2021-07-26T07:13:26.480 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:13:26.787 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672972 -lt 42949672972 2021-07-26T07:13:26.788 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2021-07-26T07:13:26.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:26.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:13:26.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:26.789 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:13:27.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: local last_scrub=2021-07-26T07:13:20.342139+0000 2021-07-26T07:13:27.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5816: TEST_periodic_scrub_replicated: ceph tell 2.0 scrub 2021-07-26T07:13:27.163 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:13:27.163 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:13:27.164 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T07:11:47.163156+0000" 2021-07-26T07:13:27.164 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:13:27.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5818: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2021-07-26T07:13:20.342139+0000 2021-07-26T07:13:27.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T07:13:27.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:13:20.342139+0000 2021-07-26T07:13:27.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:13:27.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:13:27.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:13:27.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:13:27.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:27.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:13:27.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:27.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:13:27.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:13:20.342139+0000 '>' 2021-07-26T07:13:20.342139+0000 2021-07-26T07:13:27.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:13:28.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:13:28.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:13:28.440 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:13:28.440 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:28.440 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:13:28.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:28.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:13:28.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:13:20.342139+0000 '>' 2021-07-26T07:13:20.342139+0000 2021-07-26T07:13:28.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:13:29.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:13:29.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:13:29.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:13:29.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:29.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:13:29.720 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:29.720 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:13:29.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:13:20.342139+0000 '>' 2021-07-26T07:13:20.342139+0000 2021-07-26T07:13:29.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:13:30.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:13:30.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:13:30.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:13:30.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:30.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:13:30.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:30.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:13:31.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:13:27.312636+0000 '>' 2021-07-26T07:13:20.342139+0000 2021-07-26T07:13:31.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:13:31.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2021-07-26T07:13:31.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2021-07-26T07:13:31.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: jq . 2021-07-26T07:13:31.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: grep -q POBJ 2021-07-26T07:13:31.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5827: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2021-07-26T07:13:31.684 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:13:31.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5829: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2021-07-26T07:13:31.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: flush_pg_stats 2021-07-26T07:13:31.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:13:31.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:13:32.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:13:32.143 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:13:32.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:13:32.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:32.143 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:13:32.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=120259084296 2021-07-26T07:13:32.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 120259084296 2021-07-26T07:13:32.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-120259084296' 2021-07-26T07:13:32.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:32.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:13:32.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672974 2021-07-26T07:13:32.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672974 2021-07-26T07:13:32.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-120259084296 1-42949672974' 2021-07-26T07:13:32.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:32.500 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-120259084296 2021-07-26T07:13:32.500 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:32.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:13:32.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-120259084296 2021-07-26T07:13:32.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:32.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=120259084296 2021-07-26T07:13:32.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 120259084296' 2021-07-26T07:13:32.502 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 120259084296 2021-07-26T07:13:32.503 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:13:32.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 120259084295 -lt 120259084296 2021-07-26T07:13:32.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:13:33.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:13:33.770 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:13:34.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 120259084295 -lt 120259084296 2021-07-26T07:13:34.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:13:35.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:13:35.048 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:13:35.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 120259084297 -lt 120259084296 2021-07-26T07:13:35.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:35.325 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672974 2021-07-26T07:13:35.326 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:35.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:13:35.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672974 2021-07-26T07:13:35.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:35.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672974 2021-07-26T07:13:35.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-07-26T07:13:35.329 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672974 2021-07-26T07:13:35.329 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:13:35.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672975 -lt 42949672974 2021-07-26T07:13:35.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5831: TEST_periodic_scrub_replicated: sleep 5 2021-07-26T07:13:40.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: ceph tell 2.0 scrub 2021-07-26T07:13:40.755 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:13:40.756 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:13:40.756 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T07:12:00.755807+0000" 2021-07-26T07:13:40.756 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:13:40.766 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5837: TEST_periodic_scrub_replicated: local found=false 2021-07-26T07:13:40.767 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: seq 14 -1 0 2021-07-26T07:13:40.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2021-07-26T07:13:40.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5840: TEST_periodic_scrub_replicated: sleep 1 2021-07-26T07:13:41.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5841: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2021-07-26T07:13:41.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5841: TEST_periodic_scrub_replicated: found=true 2021-07-26T07:13:41.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5841: TEST_periodic_scrub_replicated: break 2021-07-26T07:13:41.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5844: TEST_periodic_scrub_replicated: test true = true 2021-07-26T07:13:41.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5847: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2021-07-26T07:13:41.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5847: TEST_periodic_scrub_replicated: jq . 2021-07-26T07:13:41.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5847: TEST_periodic_scrub_replicated: grep -q POBJ 2021-07-26T07:13:41.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5849: TEST_periodic_scrub_replicated: flush_pg_stats 2021-07-26T07:13:41.799 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:13:41.800 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:13:42.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:13:42.082 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:13:42.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:13:42.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:42.083 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:13:42.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=120259084299 2021-07-26T07:13:42.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 120259084299 2021-07-26T07:13:42.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-120259084299' 2021-07-26T07:13:42.245 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:42.245 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:13:42.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672977 2021-07-26T07:13:42.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672977 2021-07-26T07:13:42.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-120259084299 1-42949672977' 2021-07-26T07:13:42.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:42.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-120259084299 2021-07-26T07:13:42.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:42.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:13:42.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-120259084299 2021-07-26T07:13:42.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:42.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=120259084299 2021-07-26T07:13:42.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 120259084299' 2021-07-26T07:13:42.368 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 120259084299 2021-07-26T07:13:42.368 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:13:42.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 120259084298 -lt 120259084299 2021-07-26T07:13:42.662 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:13:43.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:13:43.664 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:13:43.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 120259084298 -lt 120259084299 2021-07-26T07:13:43.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:13:44.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:13:44.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:13:45.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 120259084300 -lt 120259084299 2021-07-26T07:13:45.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:45.215 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672977 2021-07-26T07:13:45.216 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:45.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:13:45.216 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672977 2021-07-26T07:13:45.217 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:45.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672977 2021-07-26T07:13:45.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2021-07-26T07:13:45.218 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672977 2021-07-26T07:13:45.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:13:45.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672978 -lt 42949672977 2021-07-26T07:13:45.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_periodic_scrub_replicated: pg_scrub 2.0 2021-07-26T07:13:45.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: pg_scrub: local pgid=2.0 2021-07-26T07:13:45.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: pg_scrub: get_last_scrub_stamp 2.0 2021-07-26T07:13:45.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:45.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:13:45.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:45.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:13:45.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: pg_scrub: local last_scrub=2021-07-19T07:12:00.755807+0000 2021-07-26T07:13:45.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: pg_scrub: ceph pg scrub 2.0 2021-07-26T07:13:46.041 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 2.0 on osd.0 to scrub 2021-07-26T07:13:46.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: pg_scrub: wait_for_scrub 2.0 2021-07-19T07:12:00.755807+0000 2021-07-26T07:13:46.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=2.0 2021-07-26T07:13:46.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-19T07:12:00.755807+0000 2021-07-26T07:13:46.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:13:46.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:13:46.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:13:46.053 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:13:46.053 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:46.053 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:13:46.053 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:46.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:13:46.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T07:12:00.755807+0000 '>' 2021-07-19T07:12:00.755807+0000 2021-07-26T07:13:46.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:13:47.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:13:47.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:13:47.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:13:47.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:47.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:13:47.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:47.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:13:47.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T07:12:00.755807+0000 '>' 2021-07-19T07:12:00.755807+0000 2021-07-26T07:13:47.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:13:48.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:13:48.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:13:48.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:13:48.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:48.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:13:48.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:48.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:13:48.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T07:12:00.755807+0000 '>' 2021-07-19T07:12:00.755807+0000 2021-07-26T07:13:48.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:13:49.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:13:49.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:13:49.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:13:49.910 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:49.910 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:13:49.910 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:49.910 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:13:50.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T07:12:00.755807+0000 '>' 2021-07-19T07:12:00.755807+0000 2021-07-26T07:13:50.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:13:51.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:13:51.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:13:51.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2021-07-26T07:13:51.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=2.0 2021-07-26T07:13:51.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:13:51.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:13:51.225 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2021-07-26T07:13:51.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:13:46.340143+0000 '>' 2021-07-19T07:12:00.755807+0000 2021-07-26T07:13:51.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:13:51.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5852: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2021-07-26T07:13:51.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5855: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2021-07-26T07:13:51.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5855: TEST_periodic_scrub_replicated: jq . 2021-07-26T07:13:51.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5855: TEST_periodic_scrub_replicated: grep -qv POBJ 2021-07-26T07:13:51.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T07:13:51.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:13:51.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:13:51.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:13:51.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:13:51.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:13:51.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:13:51.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:13:51.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:13:51.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:13:51.584 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:13:51.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:13:51.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:13:51.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:13:51.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:13:51.586 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:13:51.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:13:51.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:13:51.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:13:51.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:13:51.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:13:51.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:13:51.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:13:51.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:13:51.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:13:51.597 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:13:51.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:13:51.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:13:51.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:13:51.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T07:13:51.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T07:13:51.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T07:13:51.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T07:13:51.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:13:51.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:13:51.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:13:51.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:13:51.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:13:51.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:13:51.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:13:51.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:13:51.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:13:51.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:13:51.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:13:51.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:13:51.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:13:51.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:13:51.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:13:51.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:13:51.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:13:51.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:13:51.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:13:51.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:13:51.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:13:51.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:13:51.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:13:51.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:13:51.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:13:51.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:13:51.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:13:51.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:13:51.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T07:13:51.614 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:13:51.615 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:13:51.615 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:13:51.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T07:13:51.616 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:13:51.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:13:51.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:13:51.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T07:13:51.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_repair_stats td/osd-scrub-repair 2021-07-26T07:13:51.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:651: TEST_repair_stats: local dir=td/osd-scrub-repair 2021-07-26T07:13:51.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:652: TEST_repair_stats: local poolname=testpool 2021-07-26T07:13:51.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:653: TEST_repair_stats: local OSDS=2 2021-07-26T07:13:51.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local OBJS=30 2021-07-26T07:13:51.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local REPAIRS=20 2021-07-26T07:13:51.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: run_mon td/osd-scrub-repair a 2021-07-26T07:13:51.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T07:13:51.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:13:51.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:13:51.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:13:51.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T07:13:51.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T07:13:51.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:13:51.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:13:51.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:13:51.978 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:13:51.979 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:13:51.979 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:13:51.979 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:13:51.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:13:52.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:13:52.223 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:13:52.223 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:13:52.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:13:52.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:13:52.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:13:52.224 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:13:52.225 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:13:52.225 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:13:52.225 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:13:52.226 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:13:52.226 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:13:52.226 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:13:52.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:13:52.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T07:13:52.319 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:13:52.319 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:13:52.319 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:13:52.319 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:13:52.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:13:52.320 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:13:52.320 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:13:52.320 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:13:52.321 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:13:52.321 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:13:52.321 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:13:52.321 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:13:52.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:13:52.322 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T07:13:52.415 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:660: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2021-07-26T07:13:52.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T07:13:52.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:13:52.416 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:13:52.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:13:52.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T07:13:52.417 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:13:53.251 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:13:53.252 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:13:53.252 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:13:53.252 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:13:53.253 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:13:53.253 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:13:53.253 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:13:53.254 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:13:53.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:13:53.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T07:13:53.277 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: expr 2 - 1 2021-07-26T07:13:53.278 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: seq 0 1 2021-07-26T07:13:53.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:13:53.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:664: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:13:53.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:13:53.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:13:53.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:13:53.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:13:53.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:13:53.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:13:53.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:13:53.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:13:53.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:13:53.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:13:53.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:13:53.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:13:53.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:13:53.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:13:53.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:13:53.288 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:13:53.288 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:13:53.288 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:13:53.288 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:13:53.289 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:13:53.289 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:13:53.289 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:13:53.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:13:53.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:13:53.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:13:53.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:13:53.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:13:53.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:13:53.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:13:53.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:13:53.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:13:53.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:13:53.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T07:13:53.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:13:53.292 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:13:53.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=233ee01e-c41e-4382-bc63-e4c055fd9389 2021-07-26T07:13:53.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 233ee01e-c41e-4382-bc63-e4c055fd9389' 2021-07-26T07:13:53.293 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 233ee01e-c41e-4382-bc63-e4c055fd9389 2021-07-26T07:13:53.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:13:53.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxYP5gDLm/ERAAfWV0IExwVlqC2SAOTOQ/Dw== 2021-07-26T07:13:53.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxYP5gDLm/ERAAfWV0IExwVlqC2SAOTOQ/Dw=="}' 2021-07-26T07:13:53.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 233ee01e-c41e-4382-bc63-e4c055fd9389 -i td/osd-scrub-repair/0/new.json 2021-07-26T07:13:53.590 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:13:53.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T07:13:53.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCxYP5gDLm/ERAAfWV0IExwVlqC2SAOTOQ/Dw== --osd-uuid 233ee01e-c41e-4382-bc63-e4c055fd9389 2021-07-26T07:13:53.627 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:53.627+0000 7fecf99130c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:53.629 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:53.629+0000 7fecf99130c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:53.630 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:53.630+0000 7fecf99130c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:53.631 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:53.630+0000 7fecf99130c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:13:53.632 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:53.630+0000 7fecf99130c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:13:53.632 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:53.630+0000 7fecf99130c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:13:53.707 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:53.707+0000 7fecf99130c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T07:13:59.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T07:13:59.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:13:59.280 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:13:59.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:13:59.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:13:59.619 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:13:59.619 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:13:59.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:13:59.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:13:59.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:13:59.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:13:59.642 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:59.640+0000 7f8b427580c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:59.643 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:59.642+0000 7f8b427580c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:59.644 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:13:59.643+0000 7f8b427580c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:59.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:13:59.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:13:59.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:13:59.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:13:59.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:13:59.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:13:59.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:13:59.894 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:13:59.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:13:59.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:14:00.160 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:00.159+0000 7f8b427580c0 -1 Falling back to public interface 2021-07-26T07:14:00.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:14:01.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:14:01.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:01.185 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:14:01.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:14:01.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:01.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:14:01.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:14:02.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:14:02.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:02.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:14:02.465 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:14:02.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:02.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:14:02.541 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:02.540+0000 7f8b427580c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:14:02.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:14:03.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:14:03.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:03.785 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:14:03.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:14:03.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:03.786 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:14:04.079 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/403796,v1:127.0.0.1:6803/403796] [v2:127.0.0.1:6804/403796,v1:127.0.0.1:6805/403796] exists,up 233ee01e-c41e-4382-bc63-e4c055fd9389 2021-07-26T07:14:04.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:14:04.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:14:04.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:14:04.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:14:04.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:664: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:14:04.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:14:04.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:14:04.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:14:04.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:14:04.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:14:04.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:14:04.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:14:04.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:14:04.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:14:04.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:14:04.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:14:04.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:14:04.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:14:04.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:14:04.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:14:04.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:14:04.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:14:04.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:14:04.089 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:14:04.089 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:14:04.090 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:14:04.090 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:14:04.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:14:04.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:14:04.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:14:04.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:14:04.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:14:04.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:14:04.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:14:04.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:14:04.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:14:04.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:14:04.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T07:14:04.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:14:04.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:14:04.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ea3e7a0e-2abf-45c7-87f4-059f701b1075 2021-07-26T07:14:04.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ea3e7a0e-2abf-45c7-87f4-059f701b1075' 2021-07-26T07:14:04.094 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 ea3e7a0e-2abf-45c7-87f4-059f701b1075 2021-07-26T07:14:04.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:14:04.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC8YP5grBPMBRAAupy7I4FO0ygNgD0jWpzaRQ== 2021-07-26T07:14:04.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC8YP5grBPMBRAAupy7I4FO0ygNgD0jWpzaRQ=="}' 2021-07-26T07:14:04.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ea3e7a0e-2abf-45c7-87f4-059f701b1075 -i td/osd-scrub-repair/1/new.json 2021-07-26T07:14:04.495 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:14:04.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T07:14:04.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQC8YP5grBPMBRAAupy7I4FO0ygNgD0jWpzaRQ== --osd-uuid ea3e7a0e-2abf-45c7-87f4-059f701b1075 2021-07-26T07:14:04.532 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:04.531+0000 7f85459c40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:04.534 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:04.533+0000 7f85459c40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:04.535 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:04.533+0000 7f85459c40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:04.536 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:04.534+0000 7f85459c40c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:14:04.536 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:04.534+0000 7f85459c40c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:14:04.536 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:04.534+0000 7f85459c40c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:14:04.602 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:04.601+0000 7f85459c40c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T07:14:10.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T07:14:10.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:14:10.047 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T07:14:10.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:14:10.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:14:10.387 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T07:14:10.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:14:10.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:14:10.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:14:10.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:14:10.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:14:10.411 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:10.409+0000 7f309e5ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:10.428 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:10.426+0000 7f309e5ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:10.428 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:10.427+0000 7f309e5ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:10.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:14:10.658 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:14:10.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:14:10.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:14:10.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:14:10.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:10.660 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:14:10.660 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:14:10.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:10.661 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:14:10.945 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:10.944+0000 7f309e5ff0c0 -1 Falling back to public interface 2021-07-26T07:14:10.963 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:14:11.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:14:11.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:11.966 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:14:11.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:14:11.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:11.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:14:12.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:14:13.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:14:13.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:13.238 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:14:13.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:14:13.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:13.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:14:13.297 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:13.295+0000 7f309e5ff0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:14:13.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:14:14.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:14:14.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:14.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:14:14.515 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:14:14.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:14.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:14:14.792 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/405195,v1:127.0.0.1:6811/405195] [v2:127.0.0.1:6812/405195,v1:127.0.0.1:6813/405195] exists,up ea3e7a0e-2abf-45c7-87f4-059f701b1075 2021-07-26T07:14:14.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:14:14.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:14:14.793 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:14:14.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: create_pool testpool 1 1 2021-07-26T07:14:14.794 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2021-07-26T07:14:15.229 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'testpool' created 2021-07-26T07:14:15.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:14:16.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:668: TEST_repair_stats: ceph osd pool set testpool size 2 2021-07-26T07:14:16.643 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 1 size to 2 2021-07-26T07:14:16.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:669: TEST_repair_stats: wait_for_clean 2021-07-26T07:14:16.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:14:16.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:14:16.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:14:16.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:14:16.655 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:14:16.655 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:14:16.656 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:14:16.916 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:14:16.917 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:14:16.917 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:14:16.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:14:16.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:14:16.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:14:16.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:14:16.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:14:17.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:14:17.018 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:14:17.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:14:17.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:14:17.019 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:14:17.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836484 2021-07-26T07:14:17.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836484 2021-07-26T07:14:17.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484' 2021-07-26T07:14:17.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:14:17.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:14:17.256 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672962 2021-07-26T07:14:17.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672962 2021-07-26T07:14:17.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484 1-42949672962' 2021-07-26T07:14:17.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:14:17.257 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836484 2021-07-26T07:14:17.258 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:14:17.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:14:17.259 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836484 2021-07-26T07:14:17.259 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:14:17.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836484 2021-07-26T07:14:17.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2021-07-26T07:14:17.260 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836484 2021-07-26T07:14:17.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:14:17.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836483 -lt 21474836484 2021-07-26T07:14:17.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:14:18.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:14:18.547 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:14:18.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836484 -lt 21474836484 2021-07-26T07:14:18.824 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:14:18.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672962 2021-07-26T07:14:18.825 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:14:18.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:14:18.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672962 2021-07-26T07:14:18.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:14:18.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672962 2021-07-26T07:14:18.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2021-07-26T07:14:18.828 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672962 2021-07-26T07:14:18.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:14:19.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672962 -lt 42949672962 2021-07-26T07:14:19.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:14:19.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:19.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:19.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:14:19.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:19.500 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:19.500 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:19.500 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:19.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:19.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:19.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:19.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:19.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:19.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:19.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:20.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:20.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T07:14:20.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T07:14:20.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T07:14:20.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:20.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:20.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:20.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:20.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:20.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:20.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:20.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:20.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:20.235 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:20.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:14:20.522 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:20.522 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:20.522 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:20.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:14:20.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:14:20.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:14:20.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: local payload=ABCDEF 2021-07-26T07:14:20.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:673: TEST_repair_stats: echo ABCDEF 2021-07-26T07:14:20.845 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: seq 1 30 2021-07-26T07:14:20.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:20.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:20.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:20.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:21.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:21.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:21.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:21.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:21.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:21.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:21.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:21.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:21.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:21.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:21.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:21.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:21.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:21.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:21.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:21.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:21.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:21.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:22.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:22.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:22.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:22.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:22.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:22.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:22.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:22.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:22.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:22.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:22.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:22.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:22.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:22.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:22.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:22.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:22.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:22.930 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:23.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:23.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:23.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:23.123 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:23.223 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:23.224 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:23.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:23.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:23.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:23.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:23.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:23.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:23.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:23.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:23.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:23.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:23.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:23.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:23.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:674: TEST_repair_stats: for i in $(seq 1 $OBJS) 2021-07-26T07:14:23.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:14:24.013 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:681: TEST_repair_stats: get_not_primary testpool obj1 2021-07-26T07:14:24.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T07:14:24.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=obj1 2021-07-26T07:14:24.015 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool obj1 2021-07-26T07:14:24.015 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T07:14:24.016 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:14:24.016 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool obj1 2021-07-26T07:14:24.016 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:14:24.280 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T07:14:24.280 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool obj1 2021-07-26T07:14:24.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T07:14:24.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:681: TEST_repair_stats: local other=0 2021-07-26T07:14:24.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:682: TEST_repair_stats: get_pg testpool obj1 2021-07-26T07:14:24.599 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=testpool 2021-07-26T07:14:24.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=obj1 2021-07-26T07:14:24.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map testpool obj1 2021-07-26T07:14:24.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:14:24.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:682: TEST_repair_stats: local pgid=1.0 2021-07-26T07:14:24.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_repair_stats: get_primary testpool obj1 2021-07-26T07:14:24.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T07:14:24.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:14:24.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool obj1 2021-07-26T07:14:24.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:14:25.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_repair_stats: local primary=1 2021-07-26T07:14:25.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:14:25.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:14:25.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:14:25.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:14:25.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:14:25.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:14:25.319 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:14:25.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:14:25.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:14:25.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:14:25.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:14:25.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:14:25.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:14:25.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:14:25.425 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: seq 1 20 2021-07-26T07:14:25.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:25.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 1 % 2 2021-07-26T07:14:25.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=1 2021-07-26T07:14:25.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2021-07-26T07:14:25.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:25.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:25.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:14:25.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:25.429 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:14:25.430 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2021-07-26T07:14:26.805 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:26.805 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:ff7b1f36:::obj1:head# 2021-07-26T07:14:27.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:27.329 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 2 % 2 2021-07-26T07:14:27.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=0 2021-07-26T07:14:27.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2021-07-26T07:14:27.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:27.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:27.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:14:27.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:27.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:14:27.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2021-07-26T07:14:28.627 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:28.628 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:104778fc:::obj2:head# 2021-07-26T07:14:29.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:29.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 3 % 2 2021-07-26T07:14:29.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=1 2021-07-26T07:14:29.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2021-07-26T07:14:29.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:29.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:29.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:14:29.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:29.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:14:29.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2021-07-26T07:14:29.914 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:29.915 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:8dd16f86:::obj3:head# 2021-07-26T07:14:30.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:30.440 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 4 % 2 2021-07-26T07:14:30.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=0 2021-07-26T07:14:30.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2021-07-26T07:14:30.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:30.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:30.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:14:30.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:30.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:14:30.444 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2021-07-26T07:14:31.167 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:31.167 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:0ee9ae15:::obj4:head# 2021-07-26T07:14:31.688 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:31.689 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 5 % 2 2021-07-26T07:14:31.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=1 2021-07-26T07:14:31.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2021-07-26T07:14:31.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:31.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:31.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:14:31.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:31.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:14:31.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2021-07-26T07:14:32.528 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:32.529 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:c52c9666:::obj5:head# 2021-07-26T07:14:33.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:33.047 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 6 % 2 2021-07-26T07:14:33.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=0 2021-07-26T07:14:33.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2021-07-26T07:14:33.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:33.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:33.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:14:33.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:33.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:14:33.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2021-07-26T07:14:33.773 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:33.773 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:233a42c1:::obj6:head# 2021-07-26T07:14:34.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:34.291 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 7 % 2 2021-07-26T07:14:34.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=1 2021-07-26T07:14:34.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2021-07-26T07:14:34.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:34.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:34.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:14:34.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:34.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:14:34.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2021-07-26T07:14:35.090 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:35.091 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:9a09113a:::obj7:head# 2021-07-26T07:14:35.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:35.615 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 8 % 2 2021-07-26T07:14:35.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=0 2021-07-26T07:14:35.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2021-07-26T07:14:35.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:35.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:35.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:14:35.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:35.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:14:35.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2021-07-26T07:14:36.488 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:36.489 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:a557efb1:::obj8:head# 2021-07-26T07:14:37.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:37.007 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 9 % 2 2021-07-26T07:14:37.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=1 2021-07-26T07:14:37.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2021-07-26T07:14:37.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:37.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:37.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:14:37.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:37.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:14:37.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2021-07-26T07:14:37.766 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:37.766 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:fe351e27:::obj9:head# 2021-07-26T07:14:38.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:38.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 10 % 2 2021-07-26T07:14:38.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=0 2021-07-26T07:14:38.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2021-07-26T07:14:38.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:38.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:38.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:14:38.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:38.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:14:38.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2021-07-26T07:14:39.019 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:39.020 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:d1337354:::obj10:head# 2021-07-26T07:14:39.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:39.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 11 % 2 2021-07-26T07:14:39.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=1 2021-07-26T07:14:39.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2021-07-26T07:14:39.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:39.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:39.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:14:39.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:39.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:14:39.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2021-07-26T07:14:40.313 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:40.314 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:eeae2a94:::obj11:head# 2021-07-26T07:14:41.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:41.082 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 12 % 2 2021-07-26T07:14:41.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=0 2021-07-26T07:14:41.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2021-07-26T07:14:41.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:41.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:41.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:14:41.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:41.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:14:41.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2021-07-26T07:14:41.781 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:41.782 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:af5f95cb:::obj12:head# 2021-07-26T07:14:42.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:42.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 13 % 2 2021-07-26T07:14:42.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=1 2021-07-26T07:14:42.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2021-07-26T07:14:42.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:42.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:42.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:14:42.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:42.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:14:42.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2021-07-26T07:14:42.988 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:42.988 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:a61ad63e:::obj13:head# 2021-07-26T07:14:43.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:43.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 14 % 2 2021-07-26T07:14:43.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=0 2021-07-26T07:14:43.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2021-07-26T07:14:43.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:43.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:43.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:14:43.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:43.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:14:43.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2021-07-26T07:14:44.257 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:44.257 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:e0a44829:::obj14:head# 2021-07-26T07:14:44.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:44.775 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 15 % 2 2021-07-26T07:14:44.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=1 2021-07-26T07:14:44.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2021-07-26T07:14:44.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:44.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:44.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:14:44.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:44.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:14:44.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2021-07-26T07:14:45.501 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:45.502 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:ab946124:::obj15:head# 2021-07-26T07:14:46.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:46.020 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 16 % 2 2021-07-26T07:14:46.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=0 2021-07-26T07:14:46.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2021-07-26T07:14:46.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:46.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:46.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:14:46.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:46.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:14:46.023 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2021-07-26T07:14:46.812 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:46.813 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:461f8b5e:::obj16:head# 2021-07-26T07:14:47.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:47.331 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 17 % 2 2021-07-26T07:14:47.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=1 2021-07-26T07:14:47.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2021-07-26T07:14:47.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:47.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:47.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:14:47.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:47.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:14:47.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2021-07-26T07:14:48.115 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:48.116 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:24fc6e92:::obj17:head# 2021-07-26T07:14:48.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:48.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 18 % 2 2021-07-26T07:14:48.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=0 2021-07-26T07:14:48.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2021-07-26T07:14:48.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:48.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:48.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:14:48.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:48.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:14:48.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2021-07-26T07:14:49.359 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:49.359 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:b9836a99:::obj18:head# 2021-07-26T07:14:49.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:49.877 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 19 % 2 2021-07-26T07:14:49.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=1 2021-07-26T07:14:49.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2021-07-26T07:14:49.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:49.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:49.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:14:49.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:49.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:14:49.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2021-07-26T07:14:50.650 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:50.651 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:00b31b6c:::obj19:head# 2021-07-26T07:14:51.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2021-07-26T07:14:51.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: expr 20 % 2 2021-07-26T07:14:51.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: OSD=0 2021-07-26T07:14:51.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2021-07-26T07:14:51.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:14:51.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:14:51.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:14:51.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:14:51.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:14:51.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2021-07-26T07:14:51.888 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:14:51.888 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:24b5611b:::obj20:head# 2021-07-26T07:14:52.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:14:52.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:14:52.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:14:52.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:14:52.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:14:52.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:14:52.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:14:52.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:14:52.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:14:52.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:14:52.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:14:52.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:14:52.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:14:52.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:14:52.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:14:52.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:14:52.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:14:52.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:14:52.669 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:14:52.670 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:14:52.670 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:14:52.670 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:14:52.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:14:52.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:14:52.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:14:52.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:14:52.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:14:52.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:14:52.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:14:52.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:14:52.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:14:52.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T07:14:52.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:14:52.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:14:52.674 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T07:14:52.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:14:52.677 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:14:52.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:14:52.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:14:52.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:14:52.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:14:52.690 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:52.689+0000 7fc4f97940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:52.692 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:52.691+0000 7fc4f97940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:52.693 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:52.692+0000 7fc4f97940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:52.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:14:52.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:14:52.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:14:52.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:14:52.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:14:52.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:52.947 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:14:52.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:14:52.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:52.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:14:53.208 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:53.207+0000 7fc4f97940c0 -1 Falling back to public interface 2021-07-26T07:14:53.254 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/405195,v1:127.0.0.1:6811/405195] [v2:127.0.0.1:6812/405195,v1:127.0.0.1:6813/405195] exists,up ea3e7a0e-2abf-45c7-87f4-059f701b1075 2021-07-26T07:14:53.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:14:53.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:14:53.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:14:53.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:14:53.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:14:53.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:14:53.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:14:53.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:14:53.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:14:53.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:14:53.264 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:14:53.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:14:53.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:14:53.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:14:53.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:14:53.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:14:53.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:14:53.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:14:53.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:14:53.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:14:53.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:14:53.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:14:53.270 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:14:53.271 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:14:53.271 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:14:53.271 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:14:53.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:14:53.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:14:53.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:14:53.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:14:53.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:14:53.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:14:53.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:14:53.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:14:53.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T07:14:53.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:14:53.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:14:53.275 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:14:53.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:14:53.277 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:14:53.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:14:53.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:14:53.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:14:53.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:14:53.280 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:53.279+0000 7fa6d1e800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:53.282 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:53.281+0000 7fa6d1e800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:53.283 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:53.282+0000 7fa6d1e800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:53.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:14:53.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:14:53.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:14:53.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:14:53.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:14:53.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:53.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:14:53.567 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:14:53.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:53.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:14:53.798 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:53.796+0000 7fa6d1e800c0 -1 Falling back to public interface 2021-07-26T07:14:53.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:14:54.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:14:54.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:54.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:14:54.833 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:14:54.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:54.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:14:55.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:14:55.752 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:55.749+0000 7fc4f97940c0 -1 osd.1 16 log_to_monitors {default=true} 2021-07-26T07:14:56.111 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:14:56.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:56.112 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:14:56.112 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:14:56.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:56.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:14:56.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:14:56.631 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:14:56.629+0000 7fa6d1e800c0 -1 osd.0 16 log_to_monitors {default=true} 2021-07-26T07:14:57.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:14:57.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:57.533 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:14:57.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:14:57.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:57.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:14:57.830 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/414705,v1:127.0.0.1:6811/414705] [v2:127.0.0.1:6812/414705,v1:127.0.0.1:6813/414705] exists,up 233ee01e-c41e-4382-bc63-e4c055fd9389 2021-07-26T07:14:57.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:14:57.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:14:57.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:14:57.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: wait_for_clean 2021-07-26T07:14:57.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:14:57.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:14:57.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:14:57.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:14:57.834 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:14:57.834 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:14:57.834 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:14:57.834 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:14:57.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:14:57.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:14:57.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:14:57.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:14:57.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:14:57.929 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:14:57.929 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:14:58.282 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:14:58.283 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:14:58.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:14:58.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:14:58.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:14:58.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=98784247810 2021-07-26T07:14:58.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 98784247810 2021-07-26T07:14:58.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-98784247810' 2021-07-26T07:14:58.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:14:58.403 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:14:58.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345922 2021-07-26T07:14:58.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345922 2021-07-26T07:14:58.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-98784247810 1-85899345922' 2021-07-26T07:14:58.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:14:58.522 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-98784247810 2021-07-26T07:14:58.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:14:58.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:14:58.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-98784247810 2021-07-26T07:14:58.523 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:14:58.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=98784247810 2021-07-26T07:14:58.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2021-07-26T07:14:58.525 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 98784247810 2021-07-26T07:14:58.525 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:14:58.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 98784247810 2021-07-26T07:14:58.790 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:14:59.792 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:14:59.793 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:15:00.069 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 98784247810 -lt 98784247810 2021-07-26T07:15:00.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:15:00.070 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-85899345922 2021-07-26T07:15:00.071 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:15:00.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:15:00.072 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-85899345922 2021-07-26T07:15:00.072 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:15:00.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345922 2021-07-26T07:15:00.073 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 85899345922' 2021-07-26T07:15:00.074 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 85899345922 2021-07-26T07:15:00.074 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:15:00.339 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345922 -lt 85899345922 2021-07-26T07:15:00.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:15:00.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:15:00.340 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:15:00.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:15:00.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:15:00.673 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:15:00.673 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:15:00.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:15:00.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:15:00.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:15:00.674 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:15:00.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:15:00.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:15:00.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:15:00.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:15:01.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:15:01.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:15:01.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:15:01.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: repair 1.0 2021-07-26T07:15:01.258 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=1.0 2021-07-26T07:15:01.259 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 1.0 2021-07-26T07:15:01.259 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:15:01.259 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:15:01.259 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:15:01.259 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:15:01.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T07:14:15.143611+0000 2021-07-26T07:15:01.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 1.0 2021-07-26T07:15:01.802 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 1.0 on osd.1 to repair 2021-07-26T07:15:01.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 1.0 2021-07-26T07:14:15.143611+0000 2021-07-26T07:15:01.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.0 2021-07-26T07:15:01.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:14:15.143611+0000 2021-07-26T07:15:01.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:15:01.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:15:01.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:15:01.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:15:01.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:15:01.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:15:01.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:15:01.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:15:02.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:14:15.143611+0000 '>' 2021-07-26T07:14:15.143611+0000 2021-07-26T07:15:02.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:15:03.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:15:03.079 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:15:03.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:15:03.081 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:15:03.081 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:15:03.081 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:15:03.081 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:15:03.354 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:14:15.143611+0000 '>' 2021-07-26T07:14:15.143611+0000 2021-07-26T07:15:03.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:15:04.355 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:15:04.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:15:04.357 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:15:04.357 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:15:04.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:15:04.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:15:04.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:15:04.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:14:15.143611+0000 '>' 2021-07-26T07:14:15.143611+0000 2021-07-26T07:15:04.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:15:05.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:15:05.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:15:05.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:15:05.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:15:05.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:15:05.653 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:15:05.654 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:15:05.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:14:15.143611+0000 '>' 2021-07-26T07:14:15.143611+0000 2021-07-26T07:15:05.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:15:06.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:15:06.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:15:06.924 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:15:06.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:15:06.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:15:06.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:15:06.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:15:07.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:14:15.143611+0000 '>' 2021-07-26T07:14:15.143611+0000 2021-07-26T07:15:07.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:15:08.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:15:08.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:15:08.198 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:15:08.199 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:15:08.199 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:15:08.199 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:15:08.199 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:15:08.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:15:01.911584+0000 '>' 2021-07-26T07:14:15.143611+0000 2021-07-26T07:15:08.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:15:08.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2021-07-26T07:15:08.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:15:08.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:15:08.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:15:08.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:15:08.473 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:15:08.473 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:15:08.473 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:15:08.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:15:08.475 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:15:08.475 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:15:08.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:15:08.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:15:08.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:15:08.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:15:08.571 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:15:08.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:15:08.833 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:15:08.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:15:08.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:15:08.834 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:15:08.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=98784247813 2021-07-26T07:15:08.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 98784247813 2021-07-26T07:15:08.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-98784247813' 2021-07-26T07:15:08.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:15:08.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:15:09.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345925 2021-07-26T07:15:09.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345925 2021-07-26T07:15:09.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-98784247813 1-85899345925' 2021-07-26T07:15:09.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:15:09.114 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-98784247813 2021-07-26T07:15:09.114 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:15:09.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:15:09.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-98784247813 2021-07-26T07:15:09.116 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:15:09.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=98784247813 2021-07-26T07:15:09.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 98784247813' 2021-07-26T07:15:09.117 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 98784247813 2021-07-26T07:15:09.117 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:15:09.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 98784247812 -lt 98784247813 2021-07-26T07:15:09.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:15:10.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:15:10.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:15:10.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 98784247813 -lt 98784247813 2021-07-26T07:15:10.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:15:10.665 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-85899345925 2021-07-26T07:15:10.665 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:15:10.665 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:15:10.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-85899345925 2021-07-26T07:15:10.666 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:15:10.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345925 2021-07-26T07:15:10.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 85899345925' 2021-07-26T07:15:10.668 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 85899345925 2021-07-26T07:15:10.668 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:15:10.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345925 -lt 85899345925 2021-07-26T07:15:10.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:15:10.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:15:10.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:15:11.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:15:11.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:15:11.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:15:11.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:15:11.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:15:11.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:15:11.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:15:11.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:15:11.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:15:11.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:15:11.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:15:11.601 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:15:11.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:15:11.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:15:11.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:15:11.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:699: TEST_repair_stats: ceph pg dump pgs 2021-07-26T07:15:12.248 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T07:15:12.248 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 30 0 0 0 0 210 0 0 30 30 active+clean 2021-07-26T07:15:05.178578+0000 16'30 24:115 [1,0] 1 [1,0] 1 16'30 2021-07-26T07:15:01.911584+0000 16'30 2021-07-26T07:15:01.911584+0000 0 2021-07-26T07:15:12.248 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T07:15:12.248 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T07:15:12.250 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T07:15:12.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: flush_pg_stats 2021-07-26T07:15:12.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:15:12.260 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:15:12.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:15:12.549 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:15:12.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:15:12.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:15:12.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:15:12.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=98784247815 2021-07-26T07:15:12.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 98784247815 2021-07-26T07:15:12.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-98784247815' 2021-07-26T07:15:12.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:15:12.684 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:15:12.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345927 2021-07-26T07:15:12.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345927 2021-07-26T07:15:12.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-98784247815 1-85899345927' 2021-07-26T07:15:12.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:15:12.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-98784247815 2021-07-26T07:15:12.803 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:15:12.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:15:12.804 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-98784247815 2021-07-26T07:15:12.805 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:15:12.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=98784247815 2021-07-26T07:15:12.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 98784247815' 2021-07-26T07:15:12.806 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 98784247815 2021-07-26T07:15:12.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:15:13.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 98784247813 -lt 98784247815 2021-07-26T07:15:13.070 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:15:14.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:15:14.072 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:15:14.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 98784247815 -lt 98784247815 2021-07-26T07:15:14.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:15:14.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-85899345927 2021-07-26T07:15:14.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:15:14.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:15:14.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-85899345927 2021-07-26T07:15:14.351 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:15:14.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345927 2021-07-26T07:15:14.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 85899345927' 2021-07-26T07:15:14.352 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 85899345927 2021-07-26T07:15:14.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:15:14.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345927 -lt 85899345927 2021-07-26T07:15:14.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: ceph pg 1.0 query 2021-07-26T07:15:14.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: jq .info.stats.stat_sum 2021-07-26T07:15:14.781 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:15:14.781 INFO:tasks.workunit.client.0.smithi071.stdout: "num_bytes": 210, 2021-07-26T07:15:14.782 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects": 30, 2021-07-26T07:15:14.782 INFO:tasks.workunit.client.0.smithi071.stdout: "num_object_clones": 0, 2021-07-26T07:15:14.782 INFO:tasks.workunit.client.0.smithi071.stdout: "num_object_copies": 60, 2021-07-26T07:15:14.782 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T07:15:14.783 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_missing": 0, 2021-07-26T07:15:14.783 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_degraded": 0, 2021-07-26T07:15:14.783 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_misplaced": 0, 2021-07-26T07:15:14.783 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_unfound": 0, 2021-07-26T07:15:14.784 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_dirty": 30, 2021-07-26T07:15:14.784 INFO:tasks.workunit.client.0.smithi071.stdout: "num_whiteouts": 0, 2021-07-26T07:15:14.784 INFO:tasks.workunit.client.0.smithi071.stdout: "num_read": 0, 2021-07-26T07:15:14.784 INFO:tasks.workunit.client.0.smithi071.stdout: "num_read_kb": 0, 2021-07-26T07:15:14.784 INFO:tasks.workunit.client.0.smithi071.stdout: "num_write": 30, 2021-07-26T07:15:14.785 INFO:tasks.workunit.client.0.smithi071.stdout: "num_write_kb": 30, 2021-07-26T07:15:14.785 INFO:tasks.workunit.client.0.smithi071.stdout: "num_scrub_errors": 0, 2021-07-26T07:15:14.785 INFO:tasks.workunit.client.0.smithi071.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T07:15:14.785 INFO:tasks.workunit.client.0.smithi071.stdout: "num_deep_scrub_errors": 0, 2021-07-26T07:15:14.786 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_recovered": 20, 2021-07-26T07:15:14.786 INFO:tasks.workunit.client.0.smithi071.stdout: "num_bytes_recovered": 140, 2021-07-26T07:15:14.786 INFO:tasks.workunit.client.0.smithi071.stdout: "num_keys_recovered": 0, 2021-07-26T07:15:14.786 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_omap": 0, 2021-07-26T07:15:14.787 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T07:15:14.787 INFO:tasks.workunit.client.0.smithi071.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T07:15:14.787 INFO:tasks.workunit.client.0.smithi071.stdout: "num_flush": 0, 2021-07-26T07:15:14.787 INFO:tasks.workunit.client.0.smithi071.stdout: "num_flush_kb": 0, 2021-07-26T07:15:14.787 INFO:tasks.workunit.client.0.smithi071.stdout: "num_evict": 0, 2021-07-26T07:15:14.788 INFO:tasks.workunit.client.0.smithi071.stdout: "num_evict_kb": 0, 2021-07-26T07:15:14.788 INFO:tasks.workunit.client.0.smithi071.stdout: "num_promote": 0, 2021-07-26T07:15:14.788 INFO:tasks.workunit.client.0.smithi071.stdout: "num_flush_mode_high": 0, 2021-07-26T07:15:14.788 INFO:tasks.workunit.client.0.smithi071.stdout: "num_flush_mode_low": 0, 2021-07-26T07:15:14.789 INFO:tasks.workunit.client.0.smithi071.stdout: "num_evict_mode_some": 0, 2021-07-26T07:15:14.789 INFO:tasks.workunit.client.0.smithi071.stdout: "num_evict_mode_full": 0, 2021-07-26T07:15:14.789 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_pinned": 0, 2021-07-26T07:15:14.789 INFO:tasks.workunit.client.0.smithi071.stdout: "num_legacy_snapsets": 0, 2021-07-26T07:15:14.789 INFO:tasks.workunit.client.0.smithi071.stdout: "num_large_omap_objects": 0, 2021-07-26T07:15:14.790 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_manifest": 0, 2021-07-26T07:15:14.790 INFO:tasks.workunit.client.0.smithi071.stdout: "num_omap_bytes": 0, 2021-07-26T07:15:14.790 INFO:tasks.workunit.client.0.smithi071.stdout: "num_omap_keys": 0, 2021-07-26T07:15:14.790 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_repaired": 20 2021-07-26T07:15:14.791 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:15:14.791 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:704: TEST_repair_stats: ceph pg 1.0 query 2021-07-26T07:15:14.792 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:704: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2021-07-26T07:15:14.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:704: TEST_repair_stats: COUNT=20 2021-07-26T07:15:14.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:705: TEST_repair_stats: test 20 = 20 2021-07-26T07:15:14.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg dump --format=json-pretty 2021-07-26T07:15:14.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2021-07-26T07:15:15.154 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:15:15.163 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:15:15.163 INFO:tasks.workunit.client.0.smithi071.stdout: "osd": 1, 2021-07-26T07:15:15.163 INFO:tasks.workunit.client.0.smithi071.stdout: "up_from": 20, 2021-07-26T07:15:15.163 INFO:tasks.workunit.client.0.smithi071.stdout: "seq": 85899345927, 2021-07-26T07:15:15.164 INFO:tasks.workunit.client.0.smithi071.stdout: "num_pgs": 1, 2021-07-26T07:15:15.164 INFO:tasks.workunit.client.0.smithi071.stdout: "num_osds": 1, 2021-07-26T07:15:15.164 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_osds": 1, 2021-07-26T07:15:15.164 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_omap_osds": 1, 2021-07-26T07:15:15.165 INFO:tasks.workunit.client.0.smithi071.stdout: "kb": 104857600, 2021-07-26T07:15:15.165 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used": 5768, 2021-07-26T07:15:15.165 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_data": 320, 2021-07-26T07:15:15.165 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_omap": 1, 2021-07-26T07:15:15.166 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_meta": 5438, 2021-07-26T07:15:15.166 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_avail": 104851832, 2021-07-26T07:15:15.166 INFO:tasks.workunit.client.0.smithi071.stdout: "statfs": { 2021-07-26T07:15:15.166 INFO:tasks.workunit.client.0.smithi071.stdout: "total": 107374182400, 2021-07-26T07:15:15.166 INFO:tasks.workunit.client.0.smithi071.stdout: "available": 107368275968, 2021-07-26T07:15:15.167 INFO:tasks.workunit.client.0.smithi071.stdout: "internally_reserved": 0, 2021-07-26T07:15:15.167 INFO:tasks.workunit.client.0.smithi071.stdout: "allocated": 327680, 2021-07-26T07:15:15.167 INFO:tasks.workunit.client.0.smithi071.stdout: "data_stored": 53197, 2021-07-26T07:15:15.167 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed": 0, 2021-07-26T07:15:15.168 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_allocated": 0, 2021-07-26T07:15:15.168 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_original": 0, 2021-07-26T07:15:15.168 INFO:tasks.workunit.client.0.smithi071.stdout: "omap_allocated": 1080, 2021-07-26T07:15:15.168 INFO:tasks.workunit.client.0.smithi071.stdout: "internal_metadata": 5569480 2021-07-26T07:15:15.168 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:15.169 INFO:tasks.workunit.client.0.smithi071.stdout: "hb_peers": [ 2021-07-26T07:15:15.169 INFO:tasks.workunit.client.0.smithi071.stdout: 0 2021-07-26T07:15:15.169 INFO:tasks.workunit.client.0.smithi071.stdout: ], 2021-07-26T07:15:15.169 INFO:tasks.workunit.client.0.smithi071.stdout: "snap_trim_queue_len": 0, 2021-07-26T07:15:15.170 INFO:tasks.workunit.client.0.smithi071.stdout: "num_snap_trimming": 0, 2021-07-26T07:15:15.170 INFO:tasks.workunit.client.0.smithi071.stdout: "num_shards_repaired": 10, 2021-07-26T07:15:15.170 INFO:tasks.workunit.client.0.smithi071.stdout: "op_queue_age_hist": { 2021-07-26T07:15:15.170 INFO:tasks.workunit.client.0.smithi071.stdout: "histogram": [], 2021-07-26T07:15:15.171 INFO:tasks.workunit.client.0.smithi071.stdout: "upper_bound": 1 2021-07-26T07:15:15.171 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:15.171 INFO:tasks.workunit.client.0.smithi071.stdout: "perf_stat": { 2021-07-26T07:15:15.171 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ms": 0, 2021-07-26T07:15:15.171 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ms": 0, 2021-07-26T07:15:15.172 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ns": 0, 2021-07-26T07:15:15.172 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ns": 0 2021-07-26T07:15:15.172 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:15.172 INFO:tasks.workunit.client.0.smithi071.stdout: "alerts": [], 2021-07-26T07:15:15.172 INFO:tasks.workunit.client.0.smithi071.stdout: "network_ping_times": [ 2021-07-26T07:15:15.173 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:15:15.173 INFO:tasks.workunit.client.0.smithi071.stdout: "osd": 0, 2021-07-26T07:15:15.173 INFO:tasks.workunit.client.0.smithi071.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2021-07-26T07:15:15.174 INFO:tasks.workunit.client.0.smithi071.stdout: "interfaces": [ 2021-07-26T07:15:15.174 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:15:15.174 INFO:tasks.workunit.client.0.smithi071.stdout: "interface": "back", 2021-07-26T07:15:15.174 INFO:tasks.workunit.client.0.smithi071.stdout: "average": { 2021-07-26T07:15:15.174 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:15:15.175 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:15:15.175 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:15:15.175 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:15.175 INFO:tasks.workunit.client.0.smithi071.stdout: "min": { 2021-07-26T07:15:15.176 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:15:15.176 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:15:15.176 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:15:15.176 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:15.176 INFO:tasks.workunit.client.0.smithi071.stdout: "max": { 2021-07-26T07:15:15.177 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:15:15.177 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:15:15.177 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:15:15.177 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:15.178 INFO:tasks.workunit.client.0.smithi071.stdout: "last": 0.176 2021-07-26T07:15:15.178 INFO:tasks.workunit.client.0.smithi071.stdout: } 2021-07-26T07:15:15.178 INFO:tasks.workunit.client.0.smithi071.stdout: ] 2021-07-26T07:15:15.178 INFO:tasks.workunit.client.0.smithi071.stdout: } 2021-07-26T07:15:15.178 INFO:tasks.workunit.client.0.smithi071.stdout: ] 2021-07-26T07:15:15.179 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:15:15.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: ceph pg dump --format=json-pretty 2021-07-26T07:15:15.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2021-07-26T07:15:15.414 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:15:15.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: COUNT=10 2021-07-26T07:15:15.424 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:709: TEST_repair_stats: expr 20 / 2 2021-07-26T07:15:15.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:709: TEST_repair_stats: test 10 = 10 2021-07-26T07:15:15.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2021-07-26T07:15:15.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2021-07-26T07:15:15.678 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:15:15.687 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:15:15.688 INFO:tasks.workunit.client.0.smithi071.stdout: "osd": 0, 2021-07-26T07:15:15.688 INFO:tasks.workunit.client.0.smithi071.stdout: "up_from": 23, 2021-07-26T07:15:15.688 INFO:tasks.workunit.client.0.smithi071.stdout: "seq": 98784247815, 2021-07-26T07:15:15.688 INFO:tasks.workunit.client.0.smithi071.stdout: "num_pgs": 1, 2021-07-26T07:15:15.689 INFO:tasks.workunit.client.0.smithi071.stdout: "num_osds": 1, 2021-07-26T07:15:15.689 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_osds": 1, 2021-07-26T07:15:15.689 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_omap_osds": 1, 2021-07-26T07:15:15.689 INFO:tasks.workunit.client.0.smithi071.stdout: "kb": 104857600, 2021-07-26T07:15:15.689 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used": 5896, 2021-07-26T07:15:15.690 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_data": 320, 2021-07-26T07:15:15.690 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_omap": 1, 2021-07-26T07:15:15.690 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_meta": 5566, 2021-07-26T07:15:15.690 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_avail": 104851704, 2021-07-26T07:15:15.691 INFO:tasks.workunit.client.0.smithi071.stdout: "statfs": { 2021-07-26T07:15:15.691 INFO:tasks.workunit.client.0.smithi071.stdout: "total": 107374182400, 2021-07-26T07:15:15.691 INFO:tasks.workunit.client.0.smithi071.stdout: "available": 107368144896, 2021-07-26T07:15:15.691 INFO:tasks.workunit.client.0.smithi071.stdout: "internally_reserved": 0, 2021-07-26T07:15:15.692 INFO:tasks.workunit.client.0.smithi071.stdout: "allocated": 327680, 2021-07-26T07:15:15.692 INFO:tasks.workunit.client.0.smithi071.stdout: "data_stored": 53197, 2021-07-26T07:15:15.692 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed": 0, 2021-07-26T07:15:15.692 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_allocated": 0, 2021-07-26T07:15:15.693 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_original": 0, 2021-07-26T07:15:15.693 INFO:tasks.workunit.client.0.smithi071.stdout: "omap_allocated": 1080, 2021-07-26T07:15:15.693 INFO:tasks.workunit.client.0.smithi071.stdout: "internal_metadata": 5700552 2021-07-26T07:15:15.693 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:15.694 INFO:tasks.workunit.client.0.smithi071.stdout: "hb_peers": [ 2021-07-26T07:15:15.694 INFO:tasks.workunit.client.0.smithi071.stdout: 1 2021-07-26T07:15:15.694 INFO:tasks.workunit.client.0.smithi071.stdout: ], 2021-07-26T07:15:15.694 INFO:tasks.workunit.client.0.smithi071.stdout: "snap_trim_queue_len": 0, 2021-07-26T07:15:15.694 INFO:tasks.workunit.client.0.smithi071.stdout: "num_snap_trimming": 0, 2021-07-26T07:15:15.695 INFO:tasks.workunit.client.0.smithi071.stdout: "num_shards_repaired": 10, 2021-07-26T07:15:15.695 INFO:tasks.workunit.client.0.smithi071.stdout: "op_queue_age_hist": { 2021-07-26T07:15:15.695 INFO:tasks.workunit.client.0.smithi071.stdout: "histogram": [], 2021-07-26T07:15:15.695 INFO:tasks.workunit.client.0.smithi071.stdout: "upper_bound": 1 2021-07-26T07:15:15.695 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:15.696 INFO:tasks.workunit.client.0.smithi071.stdout: "perf_stat": { 2021-07-26T07:15:15.696 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ms": 0, 2021-07-26T07:15:15.696 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ms": 0, 2021-07-26T07:15:15.696 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ns": 0, 2021-07-26T07:15:15.697 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ns": 0 2021-07-26T07:15:15.697 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:15.697 INFO:tasks.workunit.client.0.smithi071.stdout: "alerts": [], 2021-07-26T07:15:15.697 INFO:tasks.workunit.client.0.smithi071.stdout: "network_ping_times": [ 2021-07-26T07:15:15.697 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:15:15.698 INFO:tasks.workunit.client.0.smithi071.stdout: "osd": 1, 2021-07-26T07:15:15.698 INFO:tasks.workunit.client.0.smithi071.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2021-07-26T07:15:15.698 INFO:tasks.workunit.client.0.smithi071.stdout: "interfaces": [ 2021-07-26T07:15:15.698 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:15:15.699 INFO:tasks.workunit.client.0.smithi071.stdout: "interface": "back", 2021-07-26T07:15:15.699 INFO:tasks.workunit.client.0.smithi071.stdout: "average": { 2021-07-26T07:15:15.699 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:15:15.699 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:15:15.700 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:15:15.700 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:15.700 INFO:tasks.workunit.client.0.smithi071.stdout: "min": { 2021-07-26T07:15:15.701 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:15:15.701 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:15:15.701 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:15:15.701 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:15.701 INFO:tasks.workunit.client.0.smithi071.stdout: "max": { 2021-07-26T07:15:15.702 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:15:15.702 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:15:15.702 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:15:15.702 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:15.703 INFO:tasks.workunit.client.0.smithi071.stdout: "last": 0.144 2021-07-26T07:15:15.703 INFO:tasks.workunit.client.0.smithi071.stdout: } 2021-07-26T07:15:15.703 INFO:tasks.workunit.client.0.smithi071.stdout: ] 2021-07-26T07:15:15.703 INFO:tasks.workunit.client.0.smithi071.stdout: } 2021-07-26T07:15:15.703 INFO:tasks.workunit.client.0.smithi071.stdout: ] 2021-07-26T07:15:15.704 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:15:15.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: ceph pg dump --format=json-pretty 2021-07-26T07:15:15.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2021-07-26T07:15:15.938 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:15:15.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: COUNT=10 2021-07-26T07:15:15.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:713: TEST_repair_stats: expr 20 / 2 2021-07-26T07:15:15.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:713: TEST_repair_stats: test 10 = 10 2021-07-26T07:15:15.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2021-07-26T07:15:15.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq .pg_map.osd_stats_sum 2021-07-26T07:15:16.200 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:15:16.209 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:15:16.209 INFO:tasks.workunit.client.0.smithi071.stdout: "up_from": 0, 2021-07-26T07:15:16.209 INFO:tasks.workunit.client.0.smithi071.stdout: "seq": 0, 2021-07-26T07:15:16.210 INFO:tasks.workunit.client.0.smithi071.stdout: "num_pgs": 2, 2021-07-26T07:15:16.210 INFO:tasks.workunit.client.0.smithi071.stdout: "num_osds": 2, 2021-07-26T07:15:16.210 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_osds": 2, 2021-07-26T07:15:16.210 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_omap_osds": 2, 2021-07-26T07:15:16.211 INFO:tasks.workunit.client.0.smithi071.stdout: "kb": 209715200, 2021-07-26T07:15:16.211 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used": 11664, 2021-07-26T07:15:16.211 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_data": 640, 2021-07-26T07:15:16.212 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_omap": 2, 2021-07-26T07:15:16.212 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_meta": 11005, 2021-07-26T07:15:16.212 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_avail": 209703536, 2021-07-26T07:15:16.212 INFO:tasks.workunit.client.0.smithi071.stdout: "statfs": { 2021-07-26T07:15:16.213 INFO:tasks.workunit.client.0.smithi071.stdout: "total": 214748364800, 2021-07-26T07:15:16.213 INFO:tasks.workunit.client.0.smithi071.stdout: "available": 214736420864, 2021-07-26T07:15:16.213 INFO:tasks.workunit.client.0.smithi071.stdout: "internally_reserved": 0, 2021-07-26T07:15:16.214 INFO:tasks.workunit.client.0.smithi071.stdout: "allocated": 655360, 2021-07-26T07:15:16.214 INFO:tasks.workunit.client.0.smithi071.stdout: "data_stored": 106394, 2021-07-26T07:15:16.214 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed": 0, 2021-07-26T07:15:16.214 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_allocated": 0, 2021-07-26T07:15:16.214 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_original": 0, 2021-07-26T07:15:16.215 INFO:tasks.workunit.client.0.smithi071.stdout: "omap_allocated": 2160, 2021-07-26T07:15:16.215 INFO:tasks.workunit.client.0.smithi071.stdout: "internal_metadata": 11270032 2021-07-26T07:15:16.215 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:16.215 INFO:tasks.workunit.client.0.smithi071.stdout: "hb_peers": [], 2021-07-26T07:15:16.216 INFO:tasks.workunit.client.0.smithi071.stdout: "snap_trim_queue_len": 0, 2021-07-26T07:15:16.216 INFO:tasks.workunit.client.0.smithi071.stdout: "num_snap_trimming": 0, 2021-07-26T07:15:16.216 INFO:tasks.workunit.client.0.smithi071.stdout: "num_shards_repaired": 20, 2021-07-26T07:15:16.216 INFO:tasks.workunit.client.0.smithi071.stdout: "op_queue_age_hist": { 2021-07-26T07:15:16.217 INFO:tasks.workunit.client.0.smithi071.stdout: "histogram": [], 2021-07-26T07:15:16.217 INFO:tasks.workunit.client.0.smithi071.stdout: "upper_bound": 1 2021-07-26T07:15:16.217 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:16.217 INFO:tasks.workunit.client.0.smithi071.stdout: "perf_stat": { 2021-07-26T07:15:16.218 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ms": 0, 2021-07-26T07:15:16.218 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ms": 0, 2021-07-26T07:15:16.218 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ns": 0, 2021-07-26T07:15:16.218 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ns": 0 2021-07-26T07:15:16.219 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:15:16.219 INFO:tasks.workunit.client.0.smithi071.stdout: "alerts": [], 2021-07-26T07:15:16.219 INFO:tasks.workunit.client.0.smithi071.stdout: "network_ping_times": [] 2021-07-26T07:15:16.219 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:15:16.220 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: ceph pg dump --format=json-pretty 2021-07-26T07:15:16.220 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2021-07-26T07:15:16.463 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:15:16.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: COUNT=20 2021-07-26T07:15:16.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:717: TEST_repair_stats: test 20 = 20 2021-07-26T07:15:16.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T07:15:16.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:15:16.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:15:16.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:15:16.475 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:15:16.476 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:15:16.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:15:16.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:15:16.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:15:16.581 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:15:16.582 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:15:16.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:15:16.583 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:15:16.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:15:16.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:15:16.585 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:15:16.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:15:16.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:15:16.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:15:16.587 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:15:16.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:15:16.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:15:16.588 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:15:16.595 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:15:16.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:16.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:15:16.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:15:16.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:15:16.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:15:16.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T07:15:16.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T07:15:16.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T07:15:16.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T07:15:16.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:15:16.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:15:16.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:15:16.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:15:16.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:15:16.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:15:16.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:15:16.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:15:16.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:15:16.602 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:15:16.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:15:16.603 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:15:16.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:15:16.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:15:16.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:15:16.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:15:16.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:15:16.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:15:16.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:15:16.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:15:16.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:15:16.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:15:16.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:15:16.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:16.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:15:16.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:15:16.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:15:16.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:15:16.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T07:15:16.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:15:16.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:16.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:15:16.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T07:15:16.613 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:15:16.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:15:16.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:15:16.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T07:15:16.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_repair_stats_ec td/osd-scrub-repair 2021-07-26T07:15:16.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:721: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2021-07-26T07:15:16.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:722: TEST_repair_stats_ec: local poolname=testpool 2021-07-26T07:15:16.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:723: TEST_repair_stats_ec: local OSDS=3 2021-07-26T07:15:16.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local OBJS=30 2021-07-26T07:15:16.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local REPAIRS=26 2021-07-26T07:15:16.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local allow_overwrites=false 2021-07-26T07:15:16.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2021-07-26T07:15:16.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T07:15:16.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:15:16.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:15:16.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:15:16.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T07:15:16.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T07:15:16.966 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:15:16.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:15:16.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:15:16.967 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:15:16.968 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:16.968 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:15:16.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:15:16.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:15:17.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:15:17.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:15:17.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:15:17.228 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:15:17.228 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:15:17.228 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:15:17.228 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:15:17.229 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:15:17.229 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:15:17.230 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:15:17.230 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:17.231 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:15:17.231 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:15:17.231 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:15:17.231 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T07:15:17.326 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:15:17.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:15:17.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:15:17.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:15:17.328 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:15:17.328 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:15:17.328 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:15:17.328 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:15:17.329 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:15:17.329 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:17.329 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:15:17.330 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:15:17.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:15:17.330 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T07:15:17.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:731: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2021-07-26T07:15:17.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T07:15:17.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:15:17.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:15:17.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:15:17.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T07:15:17.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:15:18.247 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:15:18.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:15:18.248 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:15:18.249 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:15:18.250 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:18.251 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:15:18.251 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:15:18.252 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:15:18.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:15:18.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T07:15:18.271 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: expr 3 - 1 2021-07-26T07:15:18.272 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: seq 0 2 2021-07-26T07:15:18.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:15:18.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:735: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:15:18.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:15:18.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:15:18.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:15:18.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:15:18.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:15:18.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:15:18.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:15:18.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:15:18.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:15:18.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:15:18.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:15:18.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:15:18.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:15:18.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:15:18.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:15:18.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:15:18.282 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:15:18.282 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:15:18.282 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:15:18.282 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:18.283 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:15:18.283 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:15:18.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:15:18.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:15:18.284 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:15:18.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:15:18.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:15:18.285 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:15:18.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:15:18.286 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:15:18.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:15:18.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:15:18.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T07:15:18.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:15:18.288 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:15:18.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=680e278e-dd62-4245-a375-a56009ea59f1 2021-07-26T07:15:18.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 680e278e-dd62-4245-a375-a56009ea59f1' 2021-07-26T07:15:18.289 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 680e278e-dd62-4245-a375-a56009ea59f1 2021-07-26T07:15:18.289 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:15:18.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAGYf5gUe9UERAA49q19ou7ljjUbJSlcGUdUQ== 2021-07-26T07:15:18.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAGYf5gUe9UERAA49q19ou7ljjUbJSlcGUdUQ=="}' 2021-07-26T07:15:18.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 680e278e-dd62-4245-a375-a56009ea59f1 -i td/osd-scrub-repair/0/new.json 2021-07-26T07:15:18.594 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:15:18.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T07:15:18.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAGYf5gUe9UERAA49q19ou7ljjUbJSlcGUdUQ== --osd-uuid 680e278e-dd62-4245-a375-a56009ea59f1 2021-07-26T07:15:18.631 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:18.630+0000 7fe6700af0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:18.632 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:18.632+0000 7fe6700af0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:18.633 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:18.633+0000 7fe6700af0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:18.634 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:18.633+0000 7fe6700af0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:15:18.634 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:18.633+0000 7fe6700af0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:15:18.634 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:18.633+0000 7fe6700af0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:15:18.694 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:18.693+0000 7fe6700af0c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T07:15:24.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T07:15:24.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:15:24.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:15:24.244 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:15:24.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:15:24.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:15:24.583 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:15:24.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:15:24.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:15:24.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:15:24.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:15:24.605 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:24.605+0000 7efc017f90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:24.607 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:24.606+0000 7efc017f90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:24.608 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:24.607+0000 7efc017f90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:24.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:15:24.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:15:24.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:15:24.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:15:24.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:15:24.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:24.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:15:24.854 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:15:24.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:24.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:15:25.128 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:25.127+0000 7efc017f90c0 -1 Falling back to public interface 2021-07-26T07:15:25.130 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:26.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:26.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:26.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:15:26.133 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:15:26.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:26.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:15:26.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:27.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:27.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:27.402 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:15:27.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:15:27.403 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:27.404 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:15:27.521 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:27.520+0000 7efc017f90c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:15:27.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:28.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:28.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:28.713 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:15:28.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:15:28.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:28.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:15:28.985 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/418067,v1:127.0.0.1:6803/418067] [v2:127.0.0.1:6804/418067,v1:127.0.0.1:6805/418067] exists,up 680e278e-dd62-4245-a375-a56009ea59f1 2021-07-26T07:15:28.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:15:28.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:15:28.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:15:28.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:15:28.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:735: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:15:28.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:15:28.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:15:28.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:15:28.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:15:28.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:15:28.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:15:28.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:15:28.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:15:28.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:15:28.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:15:28.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:15:28.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:15:28.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:15:28.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:15:28.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:15:28.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:15:28.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:15:28.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:15:28.996 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:15:28.996 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:28.996 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:15:28.997 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:15:28.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:15:28.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:15:28.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:15:28.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:15:28.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:15:28.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:15:28.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:15:28.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:15:28.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:15:28.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:15:28.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T07:15:28.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:15:28.999 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:15:29.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f1c19742-0244-40e1-b7ed-02108ae042ed 2021-07-26T07:15:29.000 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 f1c19742-0244-40e1-b7ed-02108ae042ed 2021-07-26T07:15:29.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f1c19742-0244-40e1-b7ed-02108ae042ed' 2021-07-26T07:15:29.001 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:15:29.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARYf5ggmk2ABAACkyp94wFNF+f2UeJt5NQUA== 2021-07-26T07:15:29.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARYf5ggmk2ABAACkyp94wFNF+f2UeJt5NQUA=="}' 2021-07-26T07:15:29.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f1c19742-0244-40e1-b7ed-02108ae042ed -i td/osd-scrub-repair/1/new.json 2021-07-26T07:15:29.373 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:15:29.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T07:15:29.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQARYf5ggmk2ABAACkyp94wFNF+f2UeJt5NQUA== --osd-uuid f1c19742-0244-40e1-b7ed-02108ae042ed 2021-07-26T07:15:29.411 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:29.410+0000 7f26481a30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:29.414 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:29.414+0000 7f26481a30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:29.415 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:29.415+0000 7f26481a30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:29.416 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:29.415+0000 7f26481a30c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:15:29.417 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:29.415+0000 7f26481a30c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:15:29.417 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:29.415+0000 7f26481a30c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:15:29.493 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:29.493+0000 7f26481a30c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T07:15:34.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T07:15:34.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:15:34.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:15:34.956 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T07:15:34.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:15:35.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:15:35.295 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T07:15:35.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:15:35.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:15:35.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:15:35.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:15:35.317 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:35.317+0000 7f0b08c670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:35.319 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:35.318+0000 7f0b08c670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:35.320 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:35.319+0000 7f0b08c670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:35.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:15:35.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:15:35.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:15:35.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:15:35.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:15:35.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:35.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:15:35.598 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:15:35.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:35.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:15:35.835 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:35.835+0000 7f0b08c670c0 -1 Falling back to public interface 2021-07-26T07:15:35.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:36.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:36.906 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:36.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:15:36.907 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:15:36.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:36.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:15:37.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:38.172 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:38.171+0000 7f0b08c670c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:15:38.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:38.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:38.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:15:38.183 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:15:38.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:38.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:15:38.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:39.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:39.607 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:39.607 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:15:39.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:15:39.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:39.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:15:39.877 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/419466,v1:127.0.0.1:6811/419466] [v2:127.0.0.1:6812/419466,v1:127.0.0.1:6813/419466] exists,up f1c19742-0244-40e1-b7ed-02108ae042ed 2021-07-26T07:15:39.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:15:39.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:15:39.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:15:39.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:15:39.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:735: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:15:39.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:15:39.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:15:39.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T07:15:39.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:15:39.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:15:39.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:15:39.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:15:39.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:15:39.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:15:39.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T07:15:39.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T07:15:39.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:15:39.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:15:39.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:15:39.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:15:39.886 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:15:39.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:15:39.887 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:15:39.887 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:15:39.887 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:39.887 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:15:39.888 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:15:39.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:15:39.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:15:39.888 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:15:39.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:15:39.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:15:39.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:15:39.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:15:39.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:15:39.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:15:39.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:15:39.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T07:15:39.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T07:15:39.891 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:15:39.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd85076b-8e18-47e4-bc07-28571ebd2d86 2021-07-26T07:15:39.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 dd85076b-8e18-47e4-bc07-28571ebd2d86' 2021-07-26T07:15:39.892 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 dd85076b-8e18-47e4-bc07-28571ebd2d86 2021-07-26T07:15:39.892 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:15:39.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbYf5gs3ZaNRAAFs3knE9kZEK2bK5Oj4UAZg== 2021-07-26T07:15:39.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbYf5gs3ZaNRAAFs3knE9kZEK2bK5Oj4UAZg=="}' 2021-07-26T07:15:39.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd85076b-8e18-47e4-bc07-28571ebd2d86 -i td/osd-scrub-repair/2/new.json 2021-07-26T07:15:40.257 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:15:40.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2021-07-26T07:15:40.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAbYf5gs3ZaNRAAFs3knE9kZEK2bK5Oj4UAZg== --osd-uuid dd85076b-8e18-47e4-bc07-28571ebd2d86 2021-07-26T07:15:40.294 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:40.293+0000 7fb48bd7e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:40.315 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:40.314+0000 7fb48bd7e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:40.315 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:40.315+0000 7fb48bd7e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:40.316 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:40.315+0000 7fb48bd7e0c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T07:15:40.316 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:40.315+0000 7fb48bd7e0c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T07:15:40.317 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:40.315+0000 7fb48bd7e0c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T07:15:40.385 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:40.384+0000 7fb48bd7e0c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2021-07-26T07:15:46.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T07:15:46.201 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:15:46.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T07:15:46.202 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T07:15:46.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:15:46.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T07:15:46.538 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T07:15:46.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:15:46.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:15:46.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:15:46.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:15:46.561 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:46.561+0000 7f5b75b8c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:46.563 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:46.563+0000 7f5b75b8c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:46.564 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:46.563+0000 7f5b75b8c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:46.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T07:15:46.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:15:46.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:15:46.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:15:46.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:15:46.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:46.848 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:15:46.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:15:46.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:46.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:15:47.080 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:47.079+0000 7f5b75b8c0c0 -1 Falling back to public interface 2021-07-26T07:15:47.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:48.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:48.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:48.181 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:15:48.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:15:48.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:48.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:15:48.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:49.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:49.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:49.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:15:49.452 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:15:49.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:49.453 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:15:49.508 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:15:49.507+0000 7f5b75b8c0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T07:15:49.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:50.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:50.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:50.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:15:50.735 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:15:50.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:50.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:15:51.008 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/420865,v1:127.0.0.1:6819/420865] [v2:127.0.0.1:6820/420865,v1:127.0.0.1:6821/420865] exists,up dd85076b-8e18-47e4-bc07-28571ebd2d86 2021-07-26T07:15:51.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:15:51.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:15:51.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:15:51.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:739: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2021-07-26T07:15:51.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=testpool 2021-07-26T07:15:51.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T07:15:51.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=false 2021-07-26T07:15:51.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T07:15:51.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2021-07-26T07:15:51.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2021-07-26T07:15:51.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2021-07-26T07:15:52.255 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'testpool' created 2021-07-26T07:15:52.265 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:15:53.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' false = true ']' 2021-07-26T07:15:53.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T07:15:53.267 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:15:53.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:15:53.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:15:53.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:15:53.268 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:15:53.269 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:15:53.269 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:15:53.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:15:53.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:15:53.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:15:53.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:15:53.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:15:53.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:15:53.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:15:53.379 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:15:53.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:15:53.731 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:15:53.731 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:15:53.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:15:53.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:15:53.733 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:15:53.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836486 2021-07-26T07:15:53.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836486 2021-07-26T07:15:53.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836486' 2021-07-26T07:15:53.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:15:53.851 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:15:53.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672965 2021-07-26T07:15:53.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672965 2021-07-26T07:15:53.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2021-07-26T07:15:53.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:15:53.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:15:54.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509442 2021-07-26T07:15:54.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509442 2021-07-26T07:15:54.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509442' 2021-07-26T07:15:54.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:15:54.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836486 2021-07-26T07:15:54.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:15:54.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:15:54.091 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836486 2021-07-26T07:15:54.092 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:15:54.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836486 2021-07-26T07:15:54.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2021-07-26T07:15:54.093 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836486 2021-07-26T07:15:54.093 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:15:54.356 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836485 -lt 21474836486 2021-07-26T07:15:54.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:15:55.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:15:55.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:15:55.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836486 2021-07-26T07:15:55.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:15:55.637 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672965 2021-07-26T07:15:55.637 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:15:55.638 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:15:55.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672965 2021-07-26T07:15:55.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:15:55.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672965 2021-07-26T07:15:55.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-07-26T07:15:55.640 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672965 2021-07-26T07:15:55.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:15:55.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672966 -lt 42949672965 2021-07-26T07:15:55.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:15:55.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509442 2021-07-26T07:15:55.909 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:15:55.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:15:55.910 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509442 2021-07-26T07:15:55.911 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:15:55.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509442 2021-07-26T07:15:55.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2021-07-26T07:15:55.913 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509442 2021-07-26T07:15:55.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:15:56.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509442 -lt 64424509442 2021-07-26T07:15:56.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:15:56.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:15:56.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:15:56.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:15:56.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:15:56.564 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:15:56.565 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:15:56.565 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:15:56.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:15:56.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:15:56.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:15:56.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:15:56.865 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:15:56.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:15:56.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:15:57.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:15:57.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:15:57.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:15:57.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T07:15:57.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: local payload=ABCDEF 2021-07-26T07:15:57.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:743: TEST_repair_stats_ec: echo ABCDEF 2021-07-26T07:15:57.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: seq 1 30 2021-07-26T07:15:57.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:57.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:15:57.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:57.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:15:57.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:57.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:15:57.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:57.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:15:57.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:57.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:15:58.120 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:58.121 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:15:58.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:58.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:15:58.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:58.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:15:58.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:58.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:15:58.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:58.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:15:59.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:59.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:15:59.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:59.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:15:59.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:59.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:15:59.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:15:59.797 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:00.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:00.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:00.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:00.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:00.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:00.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:00.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:00.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:00.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:00.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:01.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:01.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:01.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:01.398 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:01.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:01.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:01.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:01.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:02.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:02.076 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:02.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:02.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:02.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:02.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:02.746 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:02.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:02.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:02.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:03.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:03.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:03.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2021-07-26T07:16:03.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2021-07-26T07:16:03.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:751: TEST_repair_stats_ec: get_not_primary testpool obj1 2021-07-26T07:16:03.661 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=testpool 2021-07-26T07:16:03.662 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=obj1 2021-07-26T07:16:03.662 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary testpool obj1 2021-07-26T07:16:03.662 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T07:16:03.662 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:16:03.663 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool obj1 2021-07-26T07:16:03.663 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:16:03.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T07:16:03.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map testpool obj1 2021-07-26T07:16:03.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T07:16:04.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:751: TEST_repair_stats_ec: local other=0 2021-07-26T07:16:04.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats_ec: get_pg testpool obj1 2021-07-26T07:16:04.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=testpool 2021-07-26T07:16:04.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=obj1 2021-07-26T07:16:04.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map testpool obj1 2021-07-26T07:16:04.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:16:04.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats_ec: local pgid=1.0 2021-07-26T07:16:04.562 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats_ec: get_primary testpool obj1 2021-07-26T07:16:04.562 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=testpool 2021-07-26T07:16:04.563 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:16:04.563 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map testpool obj1 2021-07-26T07:16:04.563 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:16:04.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats_ec: local primary=1 2021-07-26T07:16:04.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:16:04.872 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:16:04.873 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:16:04.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:16:04.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:16:04.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:16:04.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:16:04.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2021-07-26T07:16:04.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:16:04.979 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:16:04.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:16:04.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:16:04.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:16:05.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:16:05.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: seq 1 26 2021-07-26T07:16:05.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:05.085 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 1 % 2 2021-07-26T07:16:05.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=1 2021-07-26T07:16:05.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2021-07-26T07:16:05.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:05.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:05.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:16:05.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:05.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:05.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2021-07-26T07:16:06.652 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s0_head,0#1:00000000::::head#, (61) No data available 2021-07-26T07:16:06.653 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#1:ff7b1f36:::obj1:head# 2021-07-26T07:16:07.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:07.171 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 2 % 2 2021-07-26T07:16:07.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=0 2021-07-26T07:16:07.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2021-07-26T07:16:07.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:07.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:07.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:16:07.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:07.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:07.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2021-07-26T07:16:08.400 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:16:08.401 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#1:104778fc:::obj2:head# 2021-07-26T07:16:08.924 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:08.925 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 3 % 2 2021-07-26T07:16:08.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=1 2021-07-26T07:16:08.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2021-07-26T07:16:08.925 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:08.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:08.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:16:08.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:08.926 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:08.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2021-07-26T07:16:09.768 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s0_head,0#1:00000000::::head#, (61) No data available 2021-07-26T07:16:09.769 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#1:8dd16f86:::obj3:head# 2021-07-26T07:16:10.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:10.294 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 4 % 2 2021-07-26T07:16:10.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=0 2021-07-26T07:16:10.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2021-07-26T07:16:10.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:10.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:10.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:16:10.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:10.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:10.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2021-07-26T07:16:11.062 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:16:11.062 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#1:0ee9ae15:::obj4:head# 2021-07-26T07:16:11.582 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:11.583 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 5 % 2 2021-07-26T07:16:11.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=1 2021-07-26T07:16:11.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2021-07-26T07:16:11.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:11.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:11.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:16:11.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:11.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:11.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2021-07-26T07:16:12.418 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s0_head,0#1:00000000::::head#, (61) No data available 2021-07-26T07:16:12.419 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#1:c52c9666:::obj5:head# 2021-07-26T07:16:12.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:12.936 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 6 % 2 2021-07-26T07:16:12.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=0 2021-07-26T07:16:12.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2021-07-26T07:16:12.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:12.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:12.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:16:12.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:12.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:12.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2021-07-26T07:16:13.746 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:16:13.747 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#1:233a42c1:::obj6:head# 2021-07-26T07:16:14.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:14.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 7 % 2 2021-07-26T07:16:14.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=1 2021-07-26T07:16:14.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2021-07-26T07:16:14.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:14.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:14.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:16:14.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:14.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:14.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2021-07-26T07:16:15.006 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s0_head,0#1:00000000::::head#, (61) No data available 2021-07-26T07:16:15.006 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#1:9a09113a:::obj7:head# 2021-07-26T07:16:15.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:15.525 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 8 % 2 2021-07-26T07:16:15.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=0 2021-07-26T07:16:15.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2021-07-26T07:16:15.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:15.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:15.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:16:15.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:15.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:15.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2021-07-26T07:16:16.260 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:16:16.260 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#1:a557efb1:::obj8:head# 2021-07-26T07:16:16.778 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:16.779 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 9 % 2 2021-07-26T07:16:16.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=1 2021-07-26T07:16:16.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2021-07-26T07:16:16.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:16.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:16.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:16:16.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:16.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:16.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2021-07-26T07:16:17.612 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s0_head,0#1:00000000::::head#, (61) No data available 2021-07-26T07:16:17.613 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#1:fe351e27:::obj9:head# 2021-07-26T07:16:18.129 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:18.130 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 10 % 2 2021-07-26T07:16:18.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=0 2021-07-26T07:16:18.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2021-07-26T07:16:18.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:18.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:18.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:16:18.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:18.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:18.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2021-07-26T07:16:18.848 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:16:18.848 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#1:d1337354:::obj10:head# 2021-07-26T07:16:19.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:19.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 11 % 2 2021-07-26T07:16:19.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=1 2021-07-26T07:16:19.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2021-07-26T07:16:19.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:19.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:19.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:16:19.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:19.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:19.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2021-07-26T07:16:20.127 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s0_head,0#1:00000000::::head#, (61) No data available 2021-07-26T07:16:20.128 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#1:eeae2a94:::obj11:head# 2021-07-26T07:16:20.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:20.647 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 12 % 2 2021-07-26T07:16:20.648 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=0 2021-07-26T07:16:20.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2021-07-26T07:16:20.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:20.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:20.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:16:20.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:20.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:20.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2021-07-26T07:16:21.478 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:16:21.478 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#1:af5f95cb:::obj12:head# 2021-07-26T07:16:21.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:21.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 13 % 2 2021-07-26T07:16:21.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=1 2021-07-26T07:16:21.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2021-07-26T07:16:21.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:21.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:21.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:16:21.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:21.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:21.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2021-07-26T07:16:22.756 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s0_head,0#1:00000000::::head#, (61) No data available 2021-07-26T07:16:22.757 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#1:a61ad63e:::obj13:head# 2021-07-26T07:16:23.273 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:23.274 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 14 % 2 2021-07-26T07:16:23.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=0 2021-07-26T07:16:23.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2021-07-26T07:16:23.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:23.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:23.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:16:23.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:23.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:23.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2021-07-26T07:16:24.003 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:16:24.004 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#1:e0a44829:::obj14:head# 2021-07-26T07:16:24.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:24.522 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 15 % 2 2021-07-26T07:16:24.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=1 2021-07-26T07:16:24.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2021-07-26T07:16:24.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:24.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:24.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:16:24.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:24.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:24.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2021-07-26T07:16:25.330 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s0_head,0#1:00000000::::head#, (61) No data available 2021-07-26T07:16:25.331 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#1:ab946124:::obj15:head# 2021-07-26T07:16:26.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:26.098 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 16 % 2 2021-07-26T07:16:26.099 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=0 2021-07-26T07:16:26.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2021-07-26T07:16:26.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:26.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:26.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:16:26.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:26.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:26.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2021-07-26T07:16:26.826 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:16:26.826 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#1:461f8b5e:::obj16:head# 2021-07-26T07:16:27.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:27.344 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 17 % 2 2021-07-26T07:16:27.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=1 2021-07-26T07:16:27.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2021-07-26T07:16:27.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:27.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:27.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:16:27.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:27.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:27.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2021-07-26T07:16:28.111 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s0_head,0#1:00000000::::head#, (61) No data available 2021-07-26T07:16:28.112 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#1:24fc6e92:::obj17:head# 2021-07-26T07:16:28.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:28.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 18 % 2 2021-07-26T07:16:28.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=0 2021-07-26T07:16:28.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2021-07-26T07:16:28.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:28.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:28.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:16:28.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:28.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:28.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2021-07-26T07:16:29.446 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:16:29.447 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#1:b9836a99:::obj18:head# 2021-07-26T07:16:29.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:29.972 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 19 % 2 2021-07-26T07:16:29.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=1 2021-07-26T07:16:29.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2021-07-26T07:16:29.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:29.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:29.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:16:29.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:29.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:29.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2021-07-26T07:16:30.707 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s0_head,0#1:00000000::::head#, (61) No data available 2021-07-26T07:16:30.708 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#1:00b31b6c:::obj19:head# 2021-07-26T07:16:31.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:31.229 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 20 % 2 2021-07-26T07:16:31.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=0 2021-07-26T07:16:31.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2021-07-26T07:16:31.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:31.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:31.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:16:31.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:31.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:31.232 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2021-07-26T07:16:31.982 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:16:31.982 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#1:24b5611b:::obj20:head# 2021-07-26T07:16:32.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:32.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 21 % 2 2021-07-26T07:16:32.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=1 2021-07-26T07:16:32.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2021-07-26T07:16:32.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:32.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:32.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:16:32.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:32.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:32.502 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2021-07-26T07:16:33.261 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s0_head,0#1:00000000::::head#, (61) No data available 2021-07-26T07:16:33.261 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#1:1243fe1a:::obj21:head# 2021-07-26T07:16:33.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:33.782 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 22 % 2 2021-07-26T07:16:33.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=0 2021-07-26T07:16:33.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2021-07-26T07:16:33.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:33.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:33.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:16:33.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:33.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:33.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2021-07-26T07:16:34.493 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:16:34.493 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#1:c0c49def:::obj22:head# 2021-07-26T07:16:35.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:35.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 23 % 2 2021-07-26T07:16:35.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=1 2021-07-26T07:16:35.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2021-07-26T07:16:35.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:35.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:35.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:16:35.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:35.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:35.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2021-07-26T07:16:35.821 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s0_head,0#1:00000000::::head#, (61) No data available 2021-07-26T07:16:35.821 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#1:242c40e0:::obj23:head# 2021-07-26T07:16:36.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:36.346 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 24 % 2 2021-07-26T07:16:36.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=0 2021-07-26T07:16:36.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2021-07-26T07:16:36.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:36.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:36.348 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:16:36.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:36.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:36.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2021-07-26T07:16:37.111 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:16:37.112 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#1:7eacede5:::obj24:head# 2021-07-26T07:16:37.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:37.631 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 25 % 2 2021-07-26T07:16:37.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=1 2021-07-26T07:16:37.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2021-07-26T07:16:37.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:37.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:37.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T07:16:37.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:37.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:37.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2021-07-26T07:16:38.368 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s0_head,0#1:00000000::::head#, (61) No data available 2021-07-26T07:16:38.369 INFO:tasks.workunit.client.0.smithi071.stdout:remove 0#1:7109478f:::obj25:head# 2021-07-26T07:16:38.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2021-07-26T07:16:38.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: expr 26 % 2 2021-07-26T07:16:38.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: OSD=0 2021-07-26T07:16:38.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2021-07-26T07:16:38.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:16:38.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:16:38.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:16:38.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:16:38.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:38.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2021-07-26T07:16:39.663 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:16:39.664 INFO:tasks.workunit.client.0.smithi071.stdout:remove 1#1:f3c2593a:::obj26:head# 2021-07-26T07:16:40.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:16:40.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:16:40.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:16:40.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:16:40.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:16:40.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:16:40.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:16:40.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:16:40.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:16:40.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:16:40.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:16:40.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:16:40.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:16:40.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:16:40.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:16:40.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:16:40.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:16:40.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:16:40.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:16:40.190 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:16:40.190 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:16:40.190 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:16:40.190 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:16:40.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:16:40.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:16:40.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:16:40.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:16:40.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:16:40.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:16:40.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:16:40.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:16:40.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T07:16:40.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:16:40.193 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T07:16:40.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:16:40.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:16:40.195 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/1/whoami 2021-07-26T07:16:40.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:16:40.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:16:40.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:16:40.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:16:40.209 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:16:40.209+0000 7f57f87920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:16:40.211 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:16:40.211+0000 7f57f87920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:16:40.212 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:16:40.212+0000 7f57f87920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:16:40.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:16:40.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:16:40.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:16:40.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:16:40.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:16:40.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:16:40.466 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:16:40.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:16:40.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:16:40.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:16:40.736 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:16:40.736+0000 7f57f87920c0 -1 Falling back to public interface 2021-07-26T07:16:40.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:16:41.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:16:41.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:16:41.775 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:16:41.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:16:41.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:16:41.777 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:16:42.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:16:43.048 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:16:43.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:16:43.049 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:16:43.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:16:43.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:16:43.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:16:43.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:16:43.352 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:16:43.351+0000 7f57f87920c0 -1 osd.1 21 log_to_monitors {default=true} 2021-07-26T07:16:44.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:16:44.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:16:44.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:16:44.319 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:16:44.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:16:44.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:16:44.624 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [10,21) [v2:127.0.0.1:6802/432607,v1:127.0.0.1:6803/432607] [v2:127.0.0.1:6804/432607,v1:127.0.0.1:6805/432607] exists,up f1c19742-0244-40e1-b7ed-02108ae042ed 2021-07-26T07:16:44.628 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:16:44.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:16:44.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:16:44.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:16:44.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:16:44.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:16:44.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:16:44.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:16:44.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:16:44.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:16:44.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:16:44.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:16:44.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:16:44.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:16:44.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:16:44.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:16:44.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:16:44.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:16:44.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:16:44.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:16:44.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:16:44.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:16:44.634 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:16:44.634 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:16:44.634 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:16:44.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:16:44.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:16:44.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:16:44.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:16:44.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:16:44.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:16:44.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:16:44.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:16:44.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:16:44.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2021-07-26T07:16:44.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:16:44.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:16:44.638 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:16:44.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2021-07-26T07:16:44.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:16:44.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:16:44.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:16:44.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:16:44.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:16:44.651 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:16:44.649+0000 7f117cf360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:16:44.652 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:16:44.650+0000 7f117cf360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:16:44.652 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:16:44.651+0000 7f117cf360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:16:44.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:16:44.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:16:44.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:16:44.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:16:44.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:16:44.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:16:44.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:16:44.936 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:16:44.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:16:44.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:16:45.171 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:16:45.170+0000 7f117cf360c0 -1 Falling back to public interface 2021-07-26T07:16:45.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:16:46.269 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:16:46.270 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:16:46.270 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:16:46.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:16:46.271 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:16:46.272 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:16:46.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:16:47.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:16:47.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:16:47.548 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:16:47.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:16:47.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:16:47.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:16:47.602 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:16:47.600+0000 7f117cf360c0 -1 osd.0 21 log_to_monitors {default=true} 2021-07-26T07:16:47.823 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:16:48.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:16:48.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:16:48.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:16:48.826 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:16:48.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:16:48.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:16:49.095 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/433180,v1:127.0.0.1:6811/433180] [v2:127.0.0.1:6812/433180,v1:127.0.0.1:6813/433180] exists,up 680e278e-dd62-4245-a375-a56009ea59f1 2021-07-26T07:16:49.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:16:49.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:16:49.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:16:49.097 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: wait_for_clean 2021-07-26T07:16:49.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:16:49.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:16:49.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:16:49.098 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:16:49.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:16:49.099 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:16:49.099 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:16:49.099 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:16:49.100 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:16:49.100 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:16:49.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:16:49.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:16:49.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:16:49.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:16:49.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:16:49.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:16:49.484 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:16:49.484 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:16:49.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:16:49.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:49.485 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:16:49.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=124554051586 2021-07-26T07:16:49.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 124554051586 2021-07-26T07:16:49.620 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-124554051586' 2021-07-26T07:16:49.621 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:49.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:16:49.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=111669149699 2021-07-26T07:16:49.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 111669149699 2021-07-26T07:16:49.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-124554051586 1-111669149699' 2021-07-26T07:16:49.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:49.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:16:49.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509454 2021-07-26T07:16:49.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509454 2021-07-26T07:16:49.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-124554051586 1-111669149699 2-64424509454' 2021-07-26T07:16:49.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:49.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-124554051586 2021-07-26T07:16:49.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:49.857 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:16:49.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-124554051586 2021-07-26T07:16:49.859 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:49.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=124554051586 2021-07-26T07:16:49.859 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2021-07-26T07:16:49.860 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 124554051586 2021-07-26T07:16:49.860 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:16:50.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 124554051586 2021-07-26T07:16:50.125 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:16:51.126 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:16:51.127 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:16:51.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 124554051586 -lt 124554051586 2021-07-26T07:16:51.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:51.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-111669149699 2021-07-26T07:16:51.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:51.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:16:51.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-111669149699 2021-07-26T07:16:51.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:51.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=111669149699 2021-07-26T07:16:51.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 111669149699' 2021-07-26T07:16:51.450 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 111669149699 2021-07-26T07:16:51.450 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:16:51.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149699 -lt 111669149699 2021-07-26T07:16:51.756 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:51.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509454 2021-07-26T07:16:51.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:51.757 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:16:51.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509454 2021-07-26T07:16:51.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:51.759 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509454 2021-07-26T07:16:51.759 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509454 2021-07-26T07:16:51.760 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2021-07-26T07:16:51.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:16:52.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509454 -lt 64424509454 2021-07-26T07:16:52.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:16:52.025 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:52.026 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:52.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:16:52.393 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:16:52.393 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:16:52.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:16:52.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:16:52.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:16:52.394 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:16:52.395 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:16:52.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:16:52.692 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:16:52.692 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:52.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:53.025 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:16:53.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:16:53.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:16:53.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: repair 1.0 2021-07-26T07:16:53.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=1.0 2021-07-26T07:16:53.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 1.0 2021-07-26T07:16:53.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:16:53.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:16:53.027 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:16:53.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:16:53.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T07:15:52.125626+0000 2021-07-26T07:16:53.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 1.0 2021-07-26T07:16:53.595 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 1.0s0 on osd.1 to repair 2021-07-26T07:16:53.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 1.0 2021-07-26T07:15:52.125626+0000 2021-07-26T07:16:53.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.0 2021-07-26T07:16:53.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:15:52.125626+0000 2021-07-26T07:16:53.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:16:53.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:16:53.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:16:53.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:16:53.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:16:53.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:16:53.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:16:53.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:16:53.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:15:52.125626+0000 '>' 2021-07-26T07:15:52.125626+0000 2021-07-26T07:16:53.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:16:54.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:16:54.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:16:54.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:16:54.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:16:54.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:16:54.866 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:16:54.867 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:16:55.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:15:52.125626+0000 '>' 2021-07-26T07:15:52.125626+0000 2021-07-26T07:16:55.136 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:16:56.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:16:56.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:16:56.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:16:56.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:16:56.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:16:56.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:16:56.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:16:56.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:15:52.125626+0000 '>' 2021-07-26T07:15:52.125626+0000 2021-07-26T07:16:56.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:16:57.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:16:57.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:16:57.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:16:57.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:16:57.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:16:57.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:16:57.413 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:16:57.684 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:15:52.125626+0000 '>' 2021-07-26T07:15:52.125626+0000 2021-07-26T07:16:57.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:16:58.685 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:16:58.686 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:16:58.687 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:16:58.687 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:16:58.687 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:16:58.688 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:16:58.688 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:16:58.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:16:56.494947+0000 '>' 2021-07-26T07:15:52.125626+0000 2021-07-26T07:16:58.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:16:58.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2021-07-26T07:16:58.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:16:58.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:16:58.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:16:58.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:16:58.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:16:58.961 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:16:58.962 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:16:58.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:16:58.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:16:58.962 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:16:59.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:16:59.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:16:59.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:16:59.058 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:16:59.058 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:16:59.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:16:59.376 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:16:59.377 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:16:59.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:16:59.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:59.377 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:16:59.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=124554051589 2021-07-26T07:16:59.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 124554051589 2021-07-26T07:16:59.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-124554051589' 2021-07-26T07:16:59.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:59.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:16:59.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=111669149702 2021-07-26T07:16:59.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 111669149702 2021-07-26T07:16:59.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-124554051589 1-111669149702' 2021-07-26T07:16:59.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:59.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:16:59.750 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509457 2021-07-26T07:16:59.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509457 2021-07-26T07:16:59.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-124554051589 1-111669149702 2-64424509457' 2021-07-26T07:16:59.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:59.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-124554051589 2021-07-26T07:16:59.751 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:59.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:16:59.752 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-124554051589 2021-07-26T07:16:59.753 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:59.753 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=124554051589 2021-07-26T07:16:59.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2021-07-26T07:16:59.754 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 124554051589 2021-07-26T07:16:59.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:17:00.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 124554051588 -lt 124554051589 2021-07-26T07:17:00.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:17:01.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:17:01.023 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:17:01.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 124554051589 -lt 124554051589 2021-07-26T07:17:01.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:17:01.298 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-111669149702 2021-07-26T07:17:01.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:17:01.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:17:01.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-111669149702 2021-07-26T07:17:01.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:17:01.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=111669149702 2021-07-26T07:17:01.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 111669149702' 2021-07-26T07:17:01.302 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 111669149702 2021-07-26T07:17:01.302 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:17:01.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149702 -lt 111669149702 2021-07-26T07:17:01.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:17:01.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509457 2021-07-26T07:17:01.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:17:01.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:17:01.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509457 2021-07-26T07:17:01.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:17:01.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509457 2021-07-26T07:17:01.608 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2021-07-26T07:17:01.608 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509457 2021-07-26T07:17:01.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:17:01.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509457 -lt 64424509457 2021-07-26T07:17:01.897 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:17:01.897 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:01.897 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:02.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:17:02.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:02.223 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:02.223 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:02.223 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:02.223 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:02.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:02.224 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:02.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:17:02.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:02.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:02.528 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:02.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:17:02.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T07:17:02.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T07:17:02.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T07:17:02.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:17:02.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:17:02.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:17:02.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:02.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:02.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:02.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:02.951 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:02.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:02.952 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:03.226 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:17:03.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:03.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:03.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:03.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:17:03.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:17:03.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:17:03.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:17:03.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:17:03.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:17:03.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:17:03.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:17:03.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:17:03.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=10 2021-07-26T07:17:03.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 10 '!=' null 2021-07-26T07:17:03.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:17:03.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:17:03.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:17:04.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:17:04.019 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:04.020 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:04.021 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:04.021 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:04.021 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:04.021 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:04.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:04.283 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:17:04.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:04.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:04.284 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:04.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:17:04.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:17:04.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:17:04.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:17:04.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:17:04.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:17:04.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:17:04.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:17:04.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:17:04.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=10 2021-07-26T07:17:04.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 10 '!=' null 2021-07-26T07:17:04.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:17:04.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:17:04.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:17:05.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:17:05.039 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:05.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:05.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:05.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:05.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:05.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:05.040 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:05.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:17:05.324 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:05.325 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:05.325 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:05.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:17:05.672 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:17:05.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:17:05.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:769: TEST_repair_stats_ec: ceph pg dump pgs 2021-07-26T07:17:05.922 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T07:17:05.923 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 30 0 0 0 0 210 0 0 30 30 active+clean 2021-07-26T07:17:01.093773+0000 21'30 30:123 [1,0,2] 1 [1,0,2] 1 21'30 2021-07-26T07:16:56.494947+0000 21'30 2021-07-26T07:16:56.494947+0000 0 2021-07-26T07:17:05.923 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T07:17:05.923 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T07:17:05.924 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T07:17:05.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: flush_pg_stats 2021-07-26T07:17:05.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:17:05.934 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:17:06.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:17:06.199 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:17:06.199 INFO:tasks.workunit.client.0.smithi071.stderr:2' 2021-07-26T07:17:06.199 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:17:06.200 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:17:06.200 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:17:06.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=124554051591 2021-07-26T07:17:06.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 124554051591 2021-07-26T07:17:06.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-124554051591' 2021-07-26T07:17:06.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:17:06.358 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:17:06.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=111669149704 2021-07-26T07:17:06.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 111669149704 2021-07-26T07:17:06.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-124554051591 1-111669149704' 2021-07-26T07:17:06.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:17:06.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:17:06.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509460 2021-07-26T07:17:06.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509460 2021-07-26T07:17:06.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-124554051591 1-111669149704 2-64424509460' 2021-07-26T07:17:06.627 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:17:06.628 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-124554051591 2021-07-26T07:17:06.628 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:17:06.629 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:17:06.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-124554051591 2021-07-26T07:17:06.629 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:17:06.630 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=124554051591 2021-07-26T07:17:06.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2021-07-26T07:17:06.631 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 124554051591 2021-07-26T07:17:06.632 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:17:06.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 124554051590 -lt 124554051591 2021-07-26T07:17:06.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:17:07.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:17:07.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:17:08.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 124554051591 -lt 124554051591 2021-07-26T07:17:08.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:17:08.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-111669149704 2021-07-26T07:17:08.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:17:08.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:17:08.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-111669149704 2021-07-26T07:17:08.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:17:08.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=111669149704 2021-07-26T07:17:08.188 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 111669149704 2021-07-26T07:17:08.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 111669149704' 2021-07-26T07:17:08.189 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:17:08.494 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149704 -lt 111669149704 2021-07-26T07:17:08.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:17:08.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509460 2021-07-26T07:17:08.496 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:17:08.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:17:08.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509460 2021-07-26T07:17:08.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:17:08.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509460 2021-07-26T07:17:08.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2021-07-26T07:17:08.499 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509460 2021-07-26T07:17:08.499 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:17:08.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509460 -lt 64424509460 2021-07-26T07:17:08.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: ceph pg 1.0 query 2021-07-26T07:17:08.806 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: jq .info.stats.stat_sum 2021-07-26T07:17:08.964 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:17:08.964 INFO:tasks.workunit.client.0.smithi071.stdout: "num_bytes": 210, 2021-07-26T07:17:08.964 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects": 30, 2021-07-26T07:17:08.964 INFO:tasks.workunit.client.0.smithi071.stdout: "num_object_clones": 0, 2021-07-26T07:17:08.965 INFO:tasks.workunit.client.0.smithi071.stdout: "num_object_copies": 90, 2021-07-26T07:17:08.965 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T07:17:08.965 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_missing": 0, 2021-07-26T07:17:08.965 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_degraded": 0, 2021-07-26T07:17:08.965 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_misplaced": 0, 2021-07-26T07:17:08.966 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_unfound": 0, 2021-07-26T07:17:08.966 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_dirty": 30, 2021-07-26T07:17:08.966 INFO:tasks.workunit.client.0.smithi071.stdout: "num_whiteouts": 0, 2021-07-26T07:17:08.967 INFO:tasks.workunit.client.0.smithi071.stdout: "num_read": 0, 2021-07-26T07:17:08.967 INFO:tasks.workunit.client.0.smithi071.stdout: "num_read_kb": 0, 2021-07-26T07:17:08.967 INFO:tasks.workunit.client.0.smithi071.stdout: "num_write": 30, 2021-07-26T07:17:08.967 INFO:tasks.workunit.client.0.smithi071.stdout: "num_write_kb": 30, 2021-07-26T07:17:08.967 INFO:tasks.workunit.client.0.smithi071.stdout: "num_scrub_errors": 0, 2021-07-26T07:17:08.968 INFO:tasks.workunit.client.0.smithi071.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T07:17:08.968 INFO:tasks.workunit.client.0.smithi071.stdout: "num_deep_scrub_errors": 0, 2021-07-26T07:17:08.968 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_recovered": 26, 2021-07-26T07:17:08.968 INFO:tasks.workunit.client.0.smithi071.stdout: "num_bytes_recovered": 182, 2021-07-26T07:17:08.968 INFO:tasks.workunit.client.0.smithi071.stdout: "num_keys_recovered": 0, 2021-07-26T07:17:08.969 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_omap": 0, 2021-07-26T07:17:08.969 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T07:17:08.969 INFO:tasks.workunit.client.0.smithi071.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T07:17:08.969 INFO:tasks.workunit.client.0.smithi071.stdout: "num_flush": 0, 2021-07-26T07:17:08.970 INFO:tasks.workunit.client.0.smithi071.stdout: "num_flush_kb": 0, 2021-07-26T07:17:08.970 INFO:tasks.workunit.client.0.smithi071.stdout: "num_evict": 0, 2021-07-26T07:17:08.970 INFO:tasks.workunit.client.0.smithi071.stdout: "num_evict_kb": 0, 2021-07-26T07:17:08.970 INFO:tasks.workunit.client.0.smithi071.stdout: "num_promote": 0, 2021-07-26T07:17:08.971 INFO:tasks.workunit.client.0.smithi071.stdout: "num_flush_mode_high": 0, 2021-07-26T07:17:08.971 INFO:tasks.workunit.client.0.smithi071.stdout: "num_flush_mode_low": 0, 2021-07-26T07:17:08.971 INFO:tasks.workunit.client.0.smithi071.stdout: "num_evict_mode_some": 0, 2021-07-26T07:17:08.972 INFO:tasks.workunit.client.0.smithi071.stdout: "num_evict_mode_full": 0, 2021-07-26T07:17:08.972 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_pinned": 0, 2021-07-26T07:17:08.972 INFO:tasks.workunit.client.0.smithi071.stdout: "num_legacy_snapsets": 0, 2021-07-26T07:17:08.972 INFO:tasks.workunit.client.0.smithi071.stdout: "num_large_omap_objects": 0, 2021-07-26T07:17:08.973 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_manifest": 0, 2021-07-26T07:17:08.973 INFO:tasks.workunit.client.0.smithi071.stdout: "num_omap_bytes": 0, 2021-07-26T07:17:08.973 INFO:tasks.workunit.client.0.smithi071.stdout: "num_omap_keys": 0, 2021-07-26T07:17:08.974 INFO:tasks.workunit.client.0.smithi071.stdout: "num_objects_repaired": 26 2021-07-26T07:17:08.974 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:17:08.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:774: TEST_repair_stats_ec: ceph pg 1.0 query 2021-07-26T07:17:08.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:774: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2021-07-26T07:17:09.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:774: TEST_repair_stats_ec: COUNT=26 2021-07-26T07:17:09.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:775: TEST_repair_stats_ec: test 26 = 26 2021-07-26T07:17:09.087 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: expr 3 - 1 2021-07-26T07:17:09.088 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: seq 0 2 2021-07-26T07:17:09.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:17:09.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2021-07-26T07:17:09.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:779: TEST_repair_stats_ec: expr 26 / 2 2021-07-26T07:17:09.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:779: TEST_repair_stats_ec: repair=13 2021-07-26T07:17:09.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2021-07-26T07:17:09.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2021-07-26T07:17:09.340 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:17:09.349 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:17:09.350 INFO:tasks.workunit.client.0.smithi071.stdout: "osd": 0, 2021-07-26T07:17:09.350 INFO:tasks.workunit.client.0.smithi071.stdout: "up_from": 29, 2021-07-26T07:17:09.350 INFO:tasks.workunit.client.0.smithi071.stdout: "seq": 124554051592, 2021-07-26T07:17:09.350 INFO:tasks.workunit.client.0.smithi071.stdout: "num_pgs": 1, 2021-07-26T07:17:09.351 INFO:tasks.workunit.client.0.smithi071.stdout: "num_osds": 1, 2021-07-26T07:17:09.351 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_osds": 1, 2021-07-26T07:17:09.351 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_omap_osds": 1, 2021-07-26T07:17:09.351 INFO:tasks.workunit.client.0.smithi071.stdout: "kb": 104857600, 2021-07-26T07:17:09.351 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used": 5816, 2021-07-26T07:17:09.352 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_data": 368, 2021-07-26T07:17:09.352 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_omap": 1, 2021-07-26T07:17:09.352 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_meta": 5438, 2021-07-26T07:17:09.352 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_avail": 104851784, 2021-07-26T07:17:09.353 INFO:tasks.workunit.client.0.smithi071.stdout: "statfs": { 2021-07-26T07:17:09.353 INFO:tasks.workunit.client.0.smithi071.stdout: "total": 107374182400, 2021-07-26T07:17:09.354 INFO:tasks.workunit.client.0.smithi071.stdout: "available": 107368226816, 2021-07-26T07:17:09.354 INFO:tasks.workunit.client.0.smithi071.stdout: "internally_reserved": 0, 2021-07-26T07:17:09.354 INFO:tasks.workunit.client.0.smithi071.stdout: "allocated": 376832, 2021-07-26T07:17:09.355 INFO:tasks.workunit.client.0.smithi071.stdout: "data_stored": 201431, 2021-07-26T07:17:09.355 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed": 0, 2021-07-26T07:17:09.355 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_allocated": 0, 2021-07-26T07:17:09.356 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_original": 0, 2021-07-26T07:17:09.357 INFO:tasks.workunit.client.0.smithi071.stdout: "omap_allocated": 1080, 2021-07-26T07:17:09.357 INFO:tasks.workunit.client.0.smithi071.stdout: "internal_metadata": 5569480 2021-07-26T07:17:09.357 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.357 INFO:tasks.workunit.client.0.smithi071.stdout: "hb_peers": [ 2021-07-26T07:17:09.358 INFO:tasks.workunit.client.0.smithi071.stdout: 1, 2021-07-26T07:17:09.358 INFO:tasks.workunit.client.0.smithi071.stdout: 2 2021-07-26T07:17:09.358 INFO:tasks.workunit.client.0.smithi071.stdout: ], 2021-07-26T07:17:09.358 INFO:tasks.workunit.client.0.smithi071.stdout: "snap_trim_queue_len": 0, 2021-07-26T07:17:09.359 INFO:tasks.workunit.client.0.smithi071.stdout: "num_snap_trimming": 0, 2021-07-26T07:17:09.359 INFO:tasks.workunit.client.0.smithi071.stdout: "num_shards_repaired": 13, 2021-07-26T07:17:09.359 INFO:tasks.workunit.client.0.smithi071.stdout: "op_queue_age_hist": { 2021-07-26T07:17:09.360 INFO:tasks.workunit.client.0.smithi071.stdout: "histogram": [], 2021-07-26T07:17:09.360 INFO:tasks.workunit.client.0.smithi071.stdout: "upper_bound": 1 2021-07-26T07:17:09.360 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.360 INFO:tasks.workunit.client.0.smithi071.stdout: "perf_stat": { 2021-07-26T07:17:09.361 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ms": 0, 2021-07-26T07:17:09.361 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ms": 0, 2021-07-26T07:17:09.361 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ns": 0, 2021-07-26T07:17:09.361 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ns": 0 2021-07-26T07:17:09.361 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.362 INFO:tasks.workunit.client.0.smithi071.stdout: "alerts": [], 2021-07-26T07:17:09.362 INFO:tasks.workunit.client.0.smithi071.stdout: "network_ping_times": [ 2021-07-26T07:17:09.362 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:17:09.362 INFO:tasks.workunit.client.0.smithi071.stdout: "osd": 1, 2021-07-26T07:17:09.363 INFO:tasks.workunit.client.0.smithi071.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2021-07-26T07:17:09.363 INFO:tasks.workunit.client.0.smithi071.stdout: "interfaces": [ 2021-07-26T07:17:09.363 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:17:09.363 INFO:tasks.workunit.client.0.smithi071.stdout: "interface": "back", 2021-07-26T07:17:09.363 INFO:tasks.workunit.client.0.smithi071.stdout: "average": { 2021-07-26T07:17:09.364 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:09.364 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:09.364 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:09.364 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.365 INFO:tasks.workunit.client.0.smithi071.stdout: "min": { 2021-07-26T07:17:09.365 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:09.365 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:09.365 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:09.365 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.366 INFO:tasks.workunit.client.0.smithi071.stdout: "max": { 2021-07-26T07:17:09.366 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:09.366 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:09.366 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:09.367 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.367 INFO:tasks.workunit.client.0.smithi071.stdout: "last": 0.198 2021-07-26T07:17:09.367 INFO:tasks.workunit.client.0.smithi071.stdout: } 2021-07-26T07:17:09.367 INFO:tasks.workunit.client.0.smithi071.stdout: ] 2021-07-26T07:17:09.368 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.368 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:17:09.368 INFO:tasks.workunit.client.0.smithi071.stdout: "osd": 2, 2021-07-26T07:17:09.368 INFO:tasks.workunit.client.0.smithi071.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2021-07-26T07:17:09.368 INFO:tasks.workunit.client.0.smithi071.stdout: "interfaces": [ 2021-07-26T07:17:09.369 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:17:09.369 INFO:tasks.workunit.client.0.smithi071.stdout: "interface": "back", 2021-07-26T07:17:09.369 INFO:tasks.workunit.client.0.smithi071.stdout: "average": { 2021-07-26T07:17:09.369 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:09.370 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:09.370 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:09.370 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.370 INFO:tasks.workunit.client.0.smithi071.stdout: "min": { 2021-07-26T07:17:09.370 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:09.371 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:09.371 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:09.371 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.371 INFO:tasks.workunit.client.0.smithi071.stdout: "max": { 2021-07-26T07:17:09.372 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:09.372 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:09.372 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:09.372 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.372 INFO:tasks.workunit.client.0.smithi071.stdout: "last": 0.242 2021-07-26T07:17:09.373 INFO:tasks.workunit.client.0.smithi071.stdout: } 2021-07-26T07:17:09.373 INFO:tasks.workunit.client.0.smithi071.stdout: ] 2021-07-26T07:17:09.373 INFO:tasks.workunit.client.0.smithi071.stdout: } 2021-07-26T07:17:09.373 INFO:tasks.workunit.client.0.smithi071.stdout: ] 2021-07-26T07:17:09.374 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:17:09.374 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:785: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2021-07-26T07:17:09.375 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:785: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2021-07-26T07:17:09.600 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:17:09.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:785: TEST_repair_stats_ec: COUNT=13 2021-07-26T07:17:09.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:786: TEST_repair_stats_ec: test 13 = 13 2021-07-26T07:17:09.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:17:09.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: '[' 1 = 0 -o 1 = 1 ']' 2021-07-26T07:17:09.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:779: TEST_repair_stats_ec: expr 26 / 2 2021-07-26T07:17:09.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:779: TEST_repair_stats_ec: repair=13 2021-07-26T07:17:09.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2021-07-26T07:17:09.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2021-07-26T07:17:09.862 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:17:09.871 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:17:09.871 INFO:tasks.workunit.client.0.smithi071.stdout: "osd": 1, 2021-07-26T07:17:09.872 INFO:tasks.workunit.client.0.smithi071.stdout: "up_from": 26, 2021-07-26T07:17:09.872 INFO:tasks.workunit.client.0.smithi071.stdout: "seq": 111669149705, 2021-07-26T07:17:09.873 INFO:tasks.workunit.client.0.smithi071.stdout: "num_pgs": 1, 2021-07-26T07:17:09.873 INFO:tasks.workunit.client.0.smithi071.stdout: "num_osds": 1, 2021-07-26T07:17:09.873 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_osds": 1, 2021-07-26T07:17:09.874 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_omap_osds": 1, 2021-07-26T07:17:09.874 INFO:tasks.workunit.client.0.smithi071.stdout: "kb": 104857600, 2021-07-26T07:17:09.874 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used": 5944, 2021-07-26T07:17:09.874 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_data": 368, 2021-07-26T07:17:09.874 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_omap": 1, 2021-07-26T07:17:09.875 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_meta": 5566, 2021-07-26T07:17:09.875 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_avail": 104851656, 2021-07-26T07:17:09.875 INFO:tasks.workunit.client.0.smithi071.stdout: "statfs": { 2021-07-26T07:17:09.875 INFO:tasks.workunit.client.0.smithi071.stdout: "total": 107374182400, 2021-07-26T07:17:09.876 INFO:tasks.workunit.client.0.smithi071.stdout: "available": 107368095744, 2021-07-26T07:17:09.876 INFO:tasks.workunit.client.0.smithi071.stdout: "internally_reserved": 0, 2021-07-26T07:17:09.876 INFO:tasks.workunit.client.0.smithi071.stdout: "allocated": 376832, 2021-07-26T07:17:09.876 INFO:tasks.workunit.client.0.smithi071.stdout: "data_stored": 201431, 2021-07-26T07:17:09.876 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed": 0, 2021-07-26T07:17:09.877 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_allocated": 0, 2021-07-26T07:17:09.877 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_original": 0, 2021-07-26T07:17:09.877 INFO:tasks.workunit.client.0.smithi071.stdout: "omap_allocated": 1080, 2021-07-26T07:17:09.877 INFO:tasks.workunit.client.0.smithi071.stdout: "internal_metadata": 5700552 2021-07-26T07:17:09.878 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.878 INFO:tasks.workunit.client.0.smithi071.stdout: "hb_peers": [ 2021-07-26T07:17:09.878 INFO:tasks.workunit.client.0.smithi071.stdout: 0, 2021-07-26T07:17:09.878 INFO:tasks.workunit.client.0.smithi071.stdout: 2 2021-07-26T07:17:09.879 INFO:tasks.workunit.client.0.smithi071.stdout: ], 2021-07-26T07:17:09.879 INFO:tasks.workunit.client.0.smithi071.stdout: "snap_trim_queue_len": 0, 2021-07-26T07:17:09.879 INFO:tasks.workunit.client.0.smithi071.stdout: "num_snap_trimming": 0, 2021-07-26T07:17:09.880 INFO:tasks.workunit.client.0.smithi071.stdout: "num_shards_repaired": 13, 2021-07-26T07:17:09.880 INFO:tasks.workunit.client.0.smithi071.stdout: "op_queue_age_hist": { 2021-07-26T07:17:09.880 INFO:tasks.workunit.client.0.smithi071.stdout: "histogram": [], 2021-07-26T07:17:09.880 INFO:tasks.workunit.client.0.smithi071.stdout: "upper_bound": 1 2021-07-26T07:17:09.880 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.881 INFO:tasks.workunit.client.0.smithi071.stdout: "perf_stat": { 2021-07-26T07:17:09.881 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ms": 0, 2021-07-26T07:17:09.881 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ms": 0, 2021-07-26T07:17:09.881 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ns": 0, 2021-07-26T07:17:09.881 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ns": 0 2021-07-26T07:17:09.882 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.882 INFO:tasks.workunit.client.0.smithi071.stdout: "alerts": [], 2021-07-26T07:17:09.882 INFO:tasks.workunit.client.0.smithi071.stdout: "network_ping_times": [ 2021-07-26T07:17:09.882 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:17:09.883 INFO:tasks.workunit.client.0.smithi071.stdout: "osd": 0, 2021-07-26T07:17:09.883 INFO:tasks.workunit.client.0.smithi071.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2021-07-26T07:17:09.883 INFO:tasks.workunit.client.0.smithi071.stdout: "interfaces": [ 2021-07-26T07:17:09.883 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:17:09.883 INFO:tasks.workunit.client.0.smithi071.stdout: "interface": "back", 2021-07-26T07:17:09.884 INFO:tasks.workunit.client.0.smithi071.stdout: "average": { 2021-07-26T07:17:09.884 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:09.884 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:09.884 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:09.885 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.885 INFO:tasks.workunit.client.0.smithi071.stdout: "min": { 2021-07-26T07:17:09.886 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:09.886 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:09.886 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:09.887 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.887 INFO:tasks.workunit.client.0.smithi071.stdout: "max": { 2021-07-26T07:17:09.887 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:09.887 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:09.888 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:09.888 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.888 INFO:tasks.workunit.client.0.smithi071.stdout: "last": 0.202 2021-07-26T07:17:09.888 INFO:tasks.workunit.client.0.smithi071.stdout: } 2021-07-26T07:17:09.889 INFO:tasks.workunit.client.0.smithi071.stdout: ] 2021-07-26T07:17:09.889 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.889 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:17:09.889 INFO:tasks.workunit.client.0.smithi071.stdout: "osd": 2, 2021-07-26T07:17:09.889 INFO:tasks.workunit.client.0.smithi071.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2021-07-26T07:17:09.890 INFO:tasks.workunit.client.0.smithi071.stdout: "interfaces": [ 2021-07-26T07:17:09.890 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:17:09.890 INFO:tasks.workunit.client.0.smithi071.stdout: "interface": "back", 2021-07-26T07:17:09.890 INFO:tasks.workunit.client.0.smithi071.stdout: "average": { 2021-07-26T07:17:09.891 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:09.891 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:09.891 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:09.891 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.891 INFO:tasks.workunit.client.0.smithi071.stdout: "min": { 2021-07-26T07:17:09.892 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:09.892 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:09.892 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:09.892 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.892 INFO:tasks.workunit.client.0.smithi071.stdout: "max": { 2021-07-26T07:17:09.893 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:09.893 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:09.893 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:09.893 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:09.894 INFO:tasks.workunit.client.0.smithi071.stdout: "last": 0.26 2021-07-26T07:17:09.894 INFO:tasks.workunit.client.0.smithi071.stdout: } 2021-07-26T07:17:09.894 INFO:tasks.workunit.client.0.smithi071.stdout: ] 2021-07-26T07:17:09.894 INFO:tasks.workunit.client.0.smithi071.stdout: } 2021-07-26T07:17:09.894 INFO:tasks.workunit.client.0.smithi071.stdout: ] 2021-07-26T07:17:09.895 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:17:09.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:785: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2021-07-26T07:17:09.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:785: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2021-07-26T07:17:10.122 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:17:10.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:785: TEST_repair_stats_ec: COUNT=13 2021-07-26T07:17:10.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:786: TEST_repair_stats_ec: test 13 = 13 2021-07-26T07:17:10.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:17:10.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: '[' 2 = 0 -o 2 = 1 ']' 2021-07-26T07:17:10.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: repair=0 2021-07-26T07:17:10.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2021-07-26T07:17:10.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 )' 2021-07-26T07:17:10.381 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:17:10.390 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:17:10.390 INFO:tasks.workunit.client.0.smithi071.stdout: "osd": 2, 2021-07-26T07:17:10.391 INFO:tasks.workunit.client.0.smithi071.stdout: "up_from": 15, 2021-07-26T07:17:10.391 INFO:tasks.workunit.client.0.smithi071.stdout: "seq": 64424509460, 2021-07-26T07:17:10.391 INFO:tasks.workunit.client.0.smithi071.stdout: "num_pgs": 1, 2021-07-26T07:17:10.391 INFO:tasks.workunit.client.0.smithi071.stdout: "num_osds": 1, 2021-07-26T07:17:10.392 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_osds": 1, 2021-07-26T07:17:10.392 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_omap_osds": 1, 2021-07-26T07:17:10.392 INFO:tasks.workunit.client.0.smithi071.stdout: "kb": 104857600, 2021-07-26T07:17:10.392 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used": 5688, 2021-07-26T07:17:10.392 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_data": 368, 2021-07-26T07:17:10.393 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_omap": 0, 2021-07-26T07:17:10.393 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_meta": 5312, 2021-07-26T07:17:10.393 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_avail": 104851912, 2021-07-26T07:17:10.393 INFO:tasks.workunit.client.0.smithi071.stdout: "statfs": { 2021-07-26T07:17:10.393 INFO:tasks.workunit.client.0.smithi071.stdout: "total": 107374182400, 2021-07-26T07:17:10.394 INFO:tasks.workunit.client.0.smithi071.stdout: "available": 107368357888, 2021-07-26T07:17:10.394 INFO:tasks.workunit.client.0.smithi071.stdout: "internally_reserved": 0, 2021-07-26T07:17:10.394 INFO:tasks.workunit.client.0.smithi071.stdout: "allocated": 376832, 2021-07-26T07:17:10.394 INFO:tasks.workunit.client.0.smithi071.stdout: "data_stored": 201431, 2021-07-26T07:17:10.395 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed": 0, 2021-07-26T07:17:10.395 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_allocated": 0, 2021-07-26T07:17:10.395 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_original": 0, 2021-07-26T07:17:10.395 INFO:tasks.workunit.client.0.smithi071.stdout: "omap_allocated": 0, 2021-07-26T07:17:10.395 INFO:tasks.workunit.client.0.smithi071.stdout: "internal_metadata": 5439488 2021-07-26T07:17:10.396 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:10.396 INFO:tasks.workunit.client.0.smithi071.stdout: "hb_peers": [ 2021-07-26T07:17:10.396 INFO:tasks.workunit.client.0.smithi071.stdout: 0, 2021-07-26T07:17:10.396 INFO:tasks.workunit.client.0.smithi071.stdout: 1 2021-07-26T07:17:10.397 INFO:tasks.workunit.client.0.smithi071.stdout: ], 2021-07-26T07:17:10.397 INFO:tasks.workunit.client.0.smithi071.stdout: "snap_trim_queue_len": 0, 2021-07-26T07:17:10.398 INFO:tasks.workunit.client.0.smithi071.stdout: "num_snap_trimming": 0, 2021-07-26T07:17:10.399 INFO:tasks.workunit.client.0.smithi071.stdout: "num_shards_repaired": 0, 2021-07-26T07:17:10.399 INFO:tasks.workunit.client.0.smithi071.stdout: "op_queue_age_hist": { 2021-07-26T07:17:10.399 INFO:tasks.workunit.client.0.smithi071.stdout: "histogram": [], 2021-07-26T07:17:10.399 INFO:tasks.workunit.client.0.smithi071.stdout: "upper_bound": 1 2021-07-26T07:17:10.400 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:10.400 INFO:tasks.workunit.client.0.smithi071.stdout: "perf_stat": { 2021-07-26T07:17:10.400 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ms": 0, 2021-07-26T07:17:10.400 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ms": 0, 2021-07-26T07:17:10.400 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ns": 0, 2021-07-26T07:17:10.401 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ns": 0 2021-07-26T07:17:10.401 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:10.401 INFO:tasks.workunit.client.0.smithi071.stdout: "alerts": [], 2021-07-26T07:17:10.401 INFO:tasks.workunit.client.0.smithi071.stdout: "network_ping_times": [ 2021-07-26T07:17:10.402 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:17:10.402 INFO:tasks.workunit.client.0.smithi071.stdout: "osd": 0, 2021-07-26T07:17:10.402 INFO:tasks.workunit.client.0.smithi071.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2021-07-26T07:17:10.402 INFO:tasks.workunit.client.0.smithi071.stdout: "interfaces": [ 2021-07-26T07:17:10.403 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:17:10.403 INFO:tasks.workunit.client.0.smithi071.stdout: "interface": "back", 2021-07-26T07:17:10.403 INFO:tasks.workunit.client.0.smithi071.stdout: "average": { 2021-07-26T07:17:10.403 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:10.403 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:10.404 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:10.404 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:10.404 INFO:tasks.workunit.client.0.smithi071.stdout: "min": { 2021-07-26T07:17:10.405 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:10.405 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:10.405 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:10.405 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:10.406 INFO:tasks.workunit.client.0.smithi071.stdout: "max": { 2021-07-26T07:17:10.406 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:10.406 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:10.406 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:10.407 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:10.407 INFO:tasks.workunit.client.0.smithi071.stdout: "last": 0.214 2021-07-26T07:17:10.408 INFO:tasks.workunit.client.0.smithi071.stdout: } 2021-07-26T07:17:10.408 INFO:tasks.workunit.client.0.smithi071.stdout: ] 2021-07-26T07:17:10.409 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:10.409 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:17:10.410 INFO:tasks.workunit.client.0.smithi071.stdout: "osd": 1, 2021-07-26T07:17:10.410 INFO:tasks.workunit.client.0.smithi071.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2021-07-26T07:17:10.410 INFO:tasks.workunit.client.0.smithi071.stdout: "interfaces": [ 2021-07-26T07:17:10.410 INFO:tasks.workunit.client.0.smithi071.stdout: { 2021-07-26T07:17:10.411 INFO:tasks.workunit.client.0.smithi071.stdout: "interface": "back", 2021-07-26T07:17:10.411 INFO:tasks.workunit.client.0.smithi071.stdout: "average": { 2021-07-26T07:17:10.411 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:10.412 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:10.412 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:10.412 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:10.412 INFO:tasks.workunit.client.0.smithi071.stdout: "min": { 2021-07-26T07:17:10.413 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:10.413 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:10.413 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:10.413 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:10.413 INFO:tasks.workunit.client.0.smithi071.stdout: "max": { 2021-07-26T07:17:10.414 INFO:tasks.workunit.client.0.smithi071.stdout: "1min": 0, 2021-07-26T07:17:10.414 INFO:tasks.workunit.client.0.smithi071.stdout: "5min": 0, 2021-07-26T07:17:10.414 INFO:tasks.workunit.client.0.smithi071.stdout: "15min": 0 2021-07-26T07:17:10.414 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:10.415 INFO:tasks.workunit.client.0.smithi071.stdout: "last": 0.251 2021-07-26T07:17:10.415 INFO:tasks.workunit.client.0.smithi071.stdout: } 2021-07-26T07:17:10.415 INFO:tasks.workunit.client.0.smithi071.stdout: ] 2021-07-26T07:17:10.415 INFO:tasks.workunit.client.0.smithi071.stdout: } 2021-07-26T07:17:10.415 INFO:tasks.workunit.client.0.smithi071.stdout: ] 2021-07-26T07:17:10.416 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:17:10.417 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:785: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2021-07-26T07:17:10.417 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:785: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 ).num_shards_repaired' 2021-07-26T07:17:10.642 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:17:10.652 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:785: TEST_repair_stats_ec: COUNT=0 2021-07-26T07:17:10.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:786: TEST_repair_stats_ec: test 0 = 0 2021-07-26T07:17:10.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2021-07-26T07:17:10.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum 2021-07-26T07:17:10.906 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:17:10.916 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:17:10.917 INFO:tasks.workunit.client.0.smithi071.stdout: "up_from": 0, 2021-07-26T07:17:10.917 INFO:tasks.workunit.client.0.smithi071.stdout: "seq": 0, 2021-07-26T07:17:10.917 INFO:tasks.workunit.client.0.smithi071.stdout: "num_pgs": 3, 2021-07-26T07:17:10.917 INFO:tasks.workunit.client.0.smithi071.stdout: "num_osds": 3, 2021-07-26T07:17:10.918 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_osds": 3, 2021-07-26T07:17:10.918 INFO:tasks.workunit.client.0.smithi071.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T07:17:10.918 INFO:tasks.workunit.client.0.smithi071.stdout: "kb": 314572800, 2021-07-26T07:17:10.918 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used": 17448, 2021-07-26T07:17:10.919 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_data": 1104, 2021-07-26T07:17:10.919 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_omap": 2, 2021-07-26T07:17:10.919 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_used_meta": 16317, 2021-07-26T07:17:10.919 INFO:tasks.workunit.client.0.smithi071.stdout: "kb_avail": 314555352, 2021-07-26T07:17:10.920 INFO:tasks.workunit.client.0.smithi071.stdout: "statfs": { 2021-07-26T07:17:10.920 INFO:tasks.workunit.client.0.smithi071.stdout: "total": 322122547200, 2021-07-26T07:17:10.921 INFO:tasks.workunit.client.0.smithi071.stdout: "available": 322104680448, 2021-07-26T07:17:10.921 INFO:tasks.workunit.client.0.smithi071.stdout: "internally_reserved": 0, 2021-07-26T07:17:10.921 INFO:tasks.workunit.client.0.smithi071.stdout: "allocated": 1130496, 2021-07-26T07:17:10.922 INFO:tasks.workunit.client.0.smithi071.stdout: "data_stored": 604293, 2021-07-26T07:17:10.922 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed": 0, 2021-07-26T07:17:10.922 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_allocated": 0, 2021-07-26T07:17:10.922 INFO:tasks.workunit.client.0.smithi071.stdout: "data_compressed_original": 0, 2021-07-26T07:17:10.922 INFO:tasks.workunit.client.0.smithi071.stdout: "omap_allocated": 2160, 2021-07-26T07:17:10.923 INFO:tasks.workunit.client.0.smithi071.stdout: "internal_metadata": 16709520 2021-07-26T07:17:10.923 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:10.923 INFO:tasks.workunit.client.0.smithi071.stdout: "hb_peers": [], 2021-07-26T07:17:10.923 INFO:tasks.workunit.client.0.smithi071.stdout: "snap_trim_queue_len": 0, 2021-07-26T07:17:10.923 INFO:tasks.workunit.client.0.smithi071.stdout: "num_snap_trimming": 0, 2021-07-26T07:17:10.924 INFO:tasks.workunit.client.0.smithi071.stdout: "num_shards_repaired": 26, 2021-07-26T07:17:10.924 INFO:tasks.workunit.client.0.smithi071.stdout: "op_queue_age_hist": { 2021-07-26T07:17:10.924 INFO:tasks.workunit.client.0.smithi071.stdout: "histogram": [], 2021-07-26T07:17:10.924 INFO:tasks.workunit.client.0.smithi071.stdout: "upper_bound": 1 2021-07-26T07:17:10.925 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:10.925 INFO:tasks.workunit.client.0.smithi071.stdout: "perf_stat": { 2021-07-26T07:17:10.925 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ms": 0, 2021-07-26T07:17:10.925 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ms": 0, 2021-07-26T07:17:10.925 INFO:tasks.workunit.client.0.smithi071.stdout: "commit_latency_ns": 0, 2021-07-26T07:17:10.926 INFO:tasks.workunit.client.0.smithi071.stdout: "apply_latency_ns": 0 2021-07-26T07:17:10.926 INFO:tasks.workunit.client.0.smithi071.stdout: }, 2021-07-26T07:17:10.926 INFO:tasks.workunit.client.0.smithi071.stdout: "alerts": [], 2021-07-26T07:17:10.927 INFO:tasks.workunit.client.0.smithi071.stdout: "network_ping_times": [] 2021-07-26T07:17:10.927 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:17:10.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:790: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2021-07-26T07:17:10.929 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:790: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum.num_shards_repaired 2021-07-26T07:17:11.168 INFO:tasks.workunit.client.0.smithi071.stderr:dumped all 2021-07-26T07:17:11.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:790: TEST_repair_stats_ec: COUNT=26 2021-07-26T07:17:11.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:791: TEST_repair_stats_ec: test 26 = 26 2021-07-26T07:17:11.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T07:17:11.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:17:11.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:17:11.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:17:11.181 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:17:11.181 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:17:11.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:17:11.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:17:11.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:17:11.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:17:11.287 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:17:11.288 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:17:11.288 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:17:11.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:17:11.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:17:11.290 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:17:11.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:17:11.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:17:11.293 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:17:11.293 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:17:11.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:17:11.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:17:11.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:17:11.304 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:17:11.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:11.305 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:17:11.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:17:11.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:17:11.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:17:11.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T07:17:11.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T07:17:11.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T07:17:11.306 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T07:17:11.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:17:11.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:17:11.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:17:11.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:17:11.308 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:17:11.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:17:11.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:17:11.308 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:17:11.309 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:17:11.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:17:11.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:17:11.310 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:17:11.310 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:17:11.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:17:11.311 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:17:11.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:17:11.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:17:11.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:17:11.312 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:17:11.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:17:11.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:17:11.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:17:11.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:17:11.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:11.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:17:11.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:17:11.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:17:11.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:17:11.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T07:17:11.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:17:11.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:11.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:17:11.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T07:17:11.320 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:17:11.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:17:11.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:17:11.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T07:17:11.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_request_scrub_priority td/osd-scrub-repair 2021-07-26T07:17:11.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6209: TEST_request_scrub_priority: local dir=td/osd-scrub-repair 2021-07-26T07:17:11.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6210: TEST_request_scrub_priority: local poolname=psr_pool 2021-07-26T07:17:11.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6211: TEST_request_scrub_priority: local objname=POBJ 2021-07-26T07:17:11.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6212: TEST_request_scrub_priority: local OBJECTS=64 2021-07-26T07:17:11.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6213: TEST_request_scrub_priority: local PGS=8 2021-07-26T07:17:11.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6215: TEST_request_scrub_priority: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-07-26T07:17:11.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T07:17:11.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:17:11.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:17:11.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:17:11.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T07:17:11.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-07-26T07:17:11.682 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:17:11.682 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:17:11.682 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:17:11.682 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:17:11.683 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:11.683 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:17:11.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:17:11.683 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-07-26T07:17:11.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:17:11.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:17:11.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:17:11.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:17:11.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:17:11.943 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:17:11.945 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:17:11.945 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:17:11.946 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:17:11.946 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:17:11.946 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:11.946 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:17:11.947 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:17:11.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:17:11.947 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T07:17:12.043 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:17:12.043 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:17:12.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:17:12.044 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:17:12.045 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:17:12.045 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:17:12.045 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:17:12.046 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:17:12.046 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:17:12.047 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:12.047 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:17:12.047 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:17:12.048 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:17:12.048 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T07:17:12.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: run_mgr td/osd-scrub-repair x 2021-07-26T07:17:12.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T07:17:12.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:17:12.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:17:12.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:17:12.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T07:17:12.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:17:12.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:17:12.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:17:12.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:17:12.970 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:17:12.970 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:12.971 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:17:12.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:17:12.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:17:12.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:17:13.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6217: TEST_request_scrub_priority: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2021-07-26T07:17:13.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6218: TEST_request_scrub_priority: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2021-07-26T07:17:13.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2021-07-26T07:17:13.000 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:17:13.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:17:13.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:17:13.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:17:13.001 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:17:13.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:17:13.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:17:13.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:17:13.002 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:17:13.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:17:13.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:17:13.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:17:13.003 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:17:13.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:17:13.004 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:17:13.004 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:17:13.004 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:17:13.005 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:17:13.005 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:17:13.005 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:13.005 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:17:13.006 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:17:13.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:17:13.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:17:13.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:17:13.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:17:13.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:17:13.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:17:13.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:17:13.007 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:17:13.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:17:13.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:17:13.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2021-07-26T07:17:13.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:17:13.009 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:17:13.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5686c440-0f68-47a8-a33e-481a1d04c9ac 2021-07-26T07:17:13.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5686c440-0f68-47a8-a33e-481a1d04c9ac' 2021-07-26T07:17:13.009 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 5686c440-0f68-47a8-a33e-481a1d04c9ac 2021-07-26T07:17:13.010 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:17:13.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5Yf5g+ojNABAAeh1mZP95/eBg0WKZ96JgOQ== 2021-07-26T07:17:13.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5Yf5g+ojNABAAeh1mZP95/eBg0WKZ96JgOQ=="}' 2021-07-26T07:17:13.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5686c440-0f68-47a8-a33e-481a1d04c9ac -i td/osd-scrub-repair/0/new.json 2021-07-26T07:17:13.266 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:17:13.277 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T07:17:13.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQB5Yf5g+ojNABAAeh1mZP95/eBg0WKZ96JgOQ== --osd-uuid 5686c440-0f68-47a8-a33e-481a1d04c9ac 2021-07-26T07:17:13.304 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:17:13.303+0000 7fe21be160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:13.306 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:17:13.305+0000 7fe21be160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:13.306 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:17:13.306+0000 7fe21be160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:13.307 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:17:13.306+0000 7fe21be160c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:17:13.308 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:17:13.306+0000 7fe21be160c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:17:13.308 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:17:13.306+0000 7fe21be160c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:17:13.392 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:17:13.391+0000 7fe21be160c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T07:17:19.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T07:17:19.131 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:17:19.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:17:19.133 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:17:19.133 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:17:19.472 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:17:19.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:17:19.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:17:19.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2021-07-26T07:17:19.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:17:19.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:17:19.496 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:17:19.496+0000 7ff4422b80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:19.498 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:17:19.498+0000 7ff4422b80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:19.499 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:17:19.498+0000 7ff4422b80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:19.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:17:19.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:17:19.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:17:19.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:17:19.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:17:19.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:19.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:17:19.820 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:17:19.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:19.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:17:20.022 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:17:20.021+0000 7ff4422b80c0 -1 Falling back to public interface 2021-07-26T07:17:20.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:17:21.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:17:21.092 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:21.092 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:17:21.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:17:21.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:21.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:17:21.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:17:22.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:17:22.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:22.368 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:17:22.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:17:22.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:22.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:17:22.431 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:17:22.431+0000 7ff4422b80c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:17:22.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:17:23.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:17:23.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:23.675 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:17:23.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:17:23.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:23.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:17:23.965 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/436511,v1:127.0.0.1:6803/436511] [v2:127.0.0.1:6804/436511,v1:127.0.0.1:6805/436511] exists,up 5686c440-0f68-47a8-a33e-481a1d04c9ac 2021-07-26T07:17:23.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:17:23.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:17:23.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:17:23.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: create_pool psr_pool 8 8 2021-07-26T07:17:23.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 8 8 2021-07-26T07:17:24.445 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'psr_pool' created 2021-07-26T07:17:24.454 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:17:25.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: wait_for_clean 2021-07-26T07:17:25.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:17:25.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:17:25.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:17:25.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:17:25.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:17:25.458 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:17:25.458 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:17:25.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:17:25.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:17:25.460 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:17:25.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:17:25.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:17:25.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:17:25.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:17:25.558 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:17:25.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:17:25.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:17:25.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:17:25.831 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:17:25.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836482 2021-07-26T07:17:25.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836482 2021-07-26T07:17:25.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836482' 2021-07-26T07:17:25.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:17:25.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836482 2021-07-26T07:17:25.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:17:25.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:17:25.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836482 2021-07-26T07:17:25.995 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:17:25.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836482 2021-07-26T07:17:25.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836482' 2021-07-26T07:17:25.996 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836482 2021-07-26T07:17:25.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:17:26.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836481 -lt 21474836482 2021-07-26T07:17:26.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:17:27.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:17:27.301 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:17:27.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836481 -lt 21474836482 2021-07-26T07:17:27.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:17:28.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:17:28.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:17:28.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836482 -lt 21474836482 2021-07-26T07:17:28.851 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:17:28.851 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:28.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:29.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 8 == 0 2021-07-26T07:17:29.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:29.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:29.174 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:29.174 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:29.174 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:29.175 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:29.175 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:29.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=8 2021-07-26T07:17:29.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:29.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:29.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:29.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 8 = 8 2021-07-26T07:17:29.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:17:29.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:17:29.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: local osd=0 2021-07-26T07:17:29.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: add_something td/osd-scrub-repair psr_pool POBJ noscrub 2021-07-26T07:17:29.763 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:17:29.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=psr_pool 2021-07-26T07:17:29.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=POBJ 2021-07-26T07:17:29.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:17:29.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:17:29.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:17:30.230 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:17:30.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:17:30.758 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:17:30.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:17:30.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:17:30.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2021-07-26T07:17:30.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_primary psr_pool POBJ 2021-07-26T07:17:30.833 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=psr_pool 2021-07-26T07:17:30.833 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=POBJ 2021-07-26T07:17:30.833 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map psr_pool POBJ 2021-07-26T07:17:30.833 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:17:31.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local primary=0 2021-07-26T07:17:31.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6227: TEST_request_scrub_priority: get_pg psr_pool POBJ 2021-07-26T07:17:31.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=psr_pool 2021-07-26T07:17:31.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=POBJ 2021-07-26T07:17:31.102 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map psr_pool POBJ 2021-07-26T07:17:31.103 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:17:31.409 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6227: TEST_request_scrub_priority: local pg=1.1 2021-07-26T07:17:31.410 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph osd dump 2021-07-26T07:17:31.410 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: grep '^pool.*['\'']psr_pool['\'']' 2021-07-26T07:17:31.410 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: awk '{ print $2 }' 2021-07-26T07:17:31.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: poolid=1 2021-07-26T07:17:31.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6230: TEST_request_scrub_priority: local otherpgs 2021-07-26T07:17:31.677 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: expr 8 - 1 2021-07-26T07:17:31.677 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: seq 0 7 2021-07-26T07:17:31.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2021-07-26T07:17:31.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6233: TEST_request_scrub_priority: opg=1.0 2021-07-26T07:17:31.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6234: TEST_request_scrub_priority: '[' 1.0 = 1.1 ']' 2021-07-26T07:17:31.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6237: TEST_request_scrub_priority: otherpgs='1.0 ' 2021-07-26T07:17:31.680 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2021-07-26T07:17:31.680 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1 2021-07-26T07:17:31.681 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:31.681 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:31.681 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2021-07-26T07:17:31.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: local other_last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:31.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: ceph tell 1.0 scrub 1.0 2021-07-26T07:17:32.104 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:17:32.105 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:17:32.105 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T07:15:52.104166+0000" 2021-07-26T07:17:32.105 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:17:32.115 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2021-07-26T07:17:32.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6233: TEST_request_scrub_priority: opg=1.1 2021-07-26T07:17:32.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6234: TEST_request_scrub_priority: '[' 1.1 = 1.1 ']' 2021-07-26T07:17:32.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6235: TEST_request_scrub_priority: continue 2021-07-26T07:17:32.116 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2021-07-26T07:17:32.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6233: TEST_request_scrub_priority: opg=1.2 2021-07-26T07:17:32.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6234: TEST_request_scrub_priority: '[' 1.2 = 1.1 ']' 2021-07-26T07:17:32.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6237: TEST_request_scrub_priority: otherpgs='1.0 1.2 ' 2021-07-26T07:17:32.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2021-07-26T07:17:32.118 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1 2021-07-26T07:17:32.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:32.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:32.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2021-07-26T07:17:32.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: local other_last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:32.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: ceph tell 1.2 scrub 1.2 2021-07-26T07:17:32.488 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:17:32.489 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:17:32.489 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T07:15:52.488554+0000" 2021-07-26T07:17:32.489 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:17:32.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2021-07-26T07:17:32.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6233: TEST_request_scrub_priority: opg=1.3 2021-07-26T07:17:32.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6234: TEST_request_scrub_priority: '[' 1.3 = 1.1 ']' 2021-07-26T07:17:32.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6237: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 ' 2021-07-26T07:17:32.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2021-07-26T07:17:32.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1 2021-07-26T07:17:32.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:32.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:32.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2021-07-26T07:17:32.761 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: local other_last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:32.762 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: ceph tell 1.3 scrub 1.3 2021-07-26T07:17:32.872 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:17:32.872 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:17:32.873 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T07:15:52.872524+0000" 2021-07-26T07:17:32.873 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:17:32.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2021-07-26T07:17:32.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6233: TEST_request_scrub_priority: opg=1.4 2021-07-26T07:17:32.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6234: TEST_request_scrub_priority: '[' 1.4 = 1.1 ']' 2021-07-26T07:17:32.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6237: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 ' 2021-07-26T07:17:32.884 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2021-07-26T07:17:32.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1 2021-07-26T07:17:32.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:32.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:32.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2021-07-26T07:17:33.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: local other_last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:33.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: ceph tell 1.4 scrub 1.4 2021-07-26T07:17:33.257 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:17:33.257 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:17:33.257 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T07:15:53.257080+0000" 2021-07-26T07:17:33.258 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:17:33.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2021-07-26T07:17:33.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6233: TEST_request_scrub_priority: opg=1.5 2021-07-26T07:17:33.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6234: TEST_request_scrub_priority: '[' 1.5 = 1.1 ']' 2021-07-26T07:17:33.268 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6237: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 ' 2021-07-26T07:17:33.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2021-07-26T07:17:33.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1 2021-07-26T07:17:33.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:33.269 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:33.270 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2021-07-26T07:17:33.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: local other_last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:33.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: ceph tell 1.5 scrub 1.5 2021-07-26T07:17:33.638 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:17:33.639 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:17:33.639 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T07:15:53.638537+0000" 2021-07-26T07:17:33.639 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:17:33.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2021-07-26T07:17:33.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6233: TEST_request_scrub_priority: opg=1.6 2021-07-26T07:17:33.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6234: TEST_request_scrub_priority: '[' 1.6 = 1.1 ']' 2021-07-26T07:17:33.649 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6237: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 ' 2021-07-26T07:17:33.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2021-07-26T07:17:33.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1 2021-07-26T07:17:33.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:33.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:33.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2021-07-26T07:17:33.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: local other_last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:33.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: ceph tell 1.6 scrub 1.6 2021-07-26T07:17:34.023 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:17:34.023 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:17:34.023 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T07:15:54.022994+0000" 2021-07-26T07:17:34.024 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:17:34.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2021-07-26T07:17:34.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6233: TEST_request_scrub_priority: opg=1.7 2021-07-26T07:17:34.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6234: TEST_request_scrub_priority: '[' 1.7 = 1.1 ']' 2021-07-26T07:17:34.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6237: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 1.7 ' 2021-07-26T07:17:34.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2021-07-26T07:17:34.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1 2021-07-26T07:17:34.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:34.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:34.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2021-07-26T07:17:34.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: local other_last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:34.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: ceph tell 1.7 scrub 1.7 2021-07-26T07:17:34.428 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:17:34.428 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:17:34.429 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-19T07:15:54.428278+0000" 2021-07-26T07:17:34.429 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:17:34.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: sleep 15 2021-07-26T07:17:49.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6244: TEST_request_scrub_priority: flush_pg_stats 2021-07-26T07:17:49.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:17:49.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:17:49.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:17:49.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:17:49.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:17:49.720 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:17:49.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836488 2021-07-26T07:17:49.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836488 2021-07-26T07:17:49.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488' 2021-07-26T07:17:49.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:17:49.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836488 2021-07-26T07:17:49.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:17:49.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:17:49.846 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836488 2021-07-26T07:17:49.846 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:17:49.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836488 2021-07-26T07:17:49.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-07-26T07:17:49.847 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836488 2021-07-26T07:17:49.847 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:17:50.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836488 2021-07-26T07:17:50.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:17:51.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:17:51.159 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:17:51.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836488 2021-07-26T07:17:51.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:17:52.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:17:52.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:17:52.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836488 -lt 21474836488 2021-07-26T07:17:52.708 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2021-07-26T07:17:52.708 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1 2021-07-26T07:17:52.708 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:52.709 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:52.709 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2021-07-26T07:17:53.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: local last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:53.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: ceph pg scrub 1.1 2021-07-26T07:17:53.265 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 1.1 on osd.0 to scrub 2021-07-26T07:17:53.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6250: TEST_request_scrub_priority: ceph osd unset noscrub 2021-07-26T07:17:53.664 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is unset 2021-07-26T07:17:53.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6251: TEST_request_scrub_priority: ceph osd unset nodeep-scrub 2021-07-26T07:17:54.074 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is unset 2021-07-26T07:17:54.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6253: TEST_request_scrub_priority: wait_for_scrub 1.1 2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:54.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.1 2021-07-26T07:17:54.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:54.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:17:54.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:17:54.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:17:54.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2021-07-26T07:17:54.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1 2021-07-26T07:17:54.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:54.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:54.086 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2021-07-26T07:17:54.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:17:53.298478+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:54.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:17:54.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6255: TEST_request_scrub_priority: for opg in $otherpgs $pg 2021-07-26T07:17:54.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6257: TEST_request_scrub_priority: wait_for_scrub 1.0 2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:54.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.0 2021-07-26T07:17:54.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:54.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:17:54.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:17:54.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:17:54.382 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:17:54.383 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:17:54.383 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:54.383 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:54.383 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:17:54.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T07:15:52.104166+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:54.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:17:55.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:17:55.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:17:55.643 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:17:55.643 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:17:55.644 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:55.644 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:55.644 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:17:55.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T07:15:52.104166+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:55.915 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:17:56.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:17:56.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:17:56.917 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:17:56.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:17:56.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:56.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:56.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:17:57.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T07:15:52.104166+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:57.189 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:17:58.190 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:17:58.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:17:58.191 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:17:58.191 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:17:58.192 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:58.192 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:58.192 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:17:58.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T07:15:52.104166+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:58.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:17:59.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:17:59.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:17:59.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:17:59.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:17:59.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:59.466 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:59.466 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:17:59.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:17:54.264661+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:59.774 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:17:59.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6255: TEST_request_scrub_priority: for opg in $otherpgs $pg 2021-07-26T07:17:59.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6257: TEST_request_scrub_priority: wait_for_scrub 1.2 2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:59.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.2 2021-07-26T07:17:59.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:17:59.775 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:17:59.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:17:59.776 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:17:59.776 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2021-07-26T07:17:59.776 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.2 2021-07-26T07:17:59.777 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:17:59.777 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:17:59.777 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2021-07-26T07:18:00.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:17:55.215489+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:00.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:18:00.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6255: TEST_request_scrub_priority: for opg in $otherpgs $pg 2021-07-26T07:18:00.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6257: TEST_request_scrub_priority: wait_for_scrub 1.3 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:00.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.3 2021-07-26T07:18:00.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:00.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:18:00.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:18:00.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:18:00.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2021-07-26T07:18:00.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T07:18:00.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:18:00.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:18:00.039 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T07:18:00.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:17:56.258185+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:00.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:18:00.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6255: TEST_request_scrub_priority: for opg in $otherpgs $pg 2021-07-26T07:18:00.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6257: TEST_request_scrub_priority: wait_for_scrub 1.4 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:00.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.4 2021-07-26T07:18:00.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:00.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:18:00.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:18:00.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:18:00.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2021-07-26T07:18:00.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.4 2021-07-26T07:18:00.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:18:00.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:18:00.304 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2021-07-26T07:18:00.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:17:57.212766+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:00.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:18:00.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6255: TEST_request_scrub_priority: for opg in $otherpgs $pg 2021-07-26T07:18:00.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6257: TEST_request_scrub_priority: wait_for_scrub 1.5 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:00.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.5 2021-07-26T07:18:00.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:00.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:18:00.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:18:00.564 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:18:00.564 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2021-07-26T07:18:00.564 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.5 2021-07-26T07:18:00.564 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:18:00.565 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:18:00.565 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2021-07-26T07:18:00.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:17:58.226456+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:00.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:18:00.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6255: TEST_request_scrub_priority: for opg in $otherpgs $pg 2021-07-26T07:18:00.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6257: TEST_request_scrub_priority: wait_for_scrub 1.6 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:00.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.6 2021-07-26T07:18:00.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:00.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:18:00.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:18:00.828 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:18:00.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2021-07-26T07:18:00.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.6 2021-07-26T07:18:00.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:18:00.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:18:00.829 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2021-07-26T07:18:01.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T07:15:54.022994+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:01.085 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:18:02.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:18:02.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:18:02.088 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2021-07-26T07:18:02.088 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.6 2021-07-26T07:18:02.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:18:02.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:18:02.089 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2021-07-26T07:18:02.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T07:15:54.022994+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:02.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:18:03.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:18:03.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:18:03.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2021-07-26T07:18:03.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.6 2021-07-26T07:18:03.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:18:03.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:18:03.365 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2021-07-26T07:18:03.636 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T07:15:54.022994+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:03.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:18:04.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:18:04.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:18:04.640 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2021-07-26T07:18:04.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.6 2021-07-26T07:18:04.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:18:04.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:18:04.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2021-07-26T07:18:04.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-19T07:15:54.022994+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:04.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:18:05.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:18:05.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:18:05.912 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2021-07-26T07:18:05.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.6 2021-07-26T07:18:05.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:18:05.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:18:05.913 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2021-07-26T07:18:06.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:17:59.217040+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:06.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:18:06.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6255: TEST_request_scrub_priority: for opg in $otherpgs $pg 2021-07-26T07:18:06.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6257: TEST_request_scrub_priority: wait_for_scrub 1.7 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:06.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.7 2021-07-26T07:18:06.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:06.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:18:06.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:18:06.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:18:06.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2021-07-26T07:18:06.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.7 2021-07-26T07:18:06.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:18:06.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:18:06.187 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2021-07-26T07:18:06.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:18:00.186937+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:06.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:18:06.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6255: TEST_request_scrub_priority: for opg in $otherpgs $pg 2021-07-26T07:18:06.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6257: TEST_request_scrub_priority: wait_for_scrub 1.1 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:06.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.1 2021-07-26T07:18:06.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:06.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:18:06.447 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:18:06.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:18:06.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2021-07-26T07:18:06.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.1 2021-07-26T07:18:06.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:18:06.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:18:06.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2021-07-26T07:18:06.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:17:53.298478+0000 '>' 2021-07-26T07:17:24.299405+0000 2021-07-26T07:18:06.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:18:06.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6261: TEST_request_scrub_priority: grep 'log_channel.*scrub ok' td/osd-scrub-repair/osd.0.log 2021-07-26T07:18:06.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6261: TEST_request_scrub_priority: grep -v purged_snaps 2021-07-26T07:18:06.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6261: TEST_request_scrub_priority: head -1 2021-07-26T07:18:06.707 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6261: TEST_request_scrub_priority: sed 's/.*[[]DBG[]]//' 2021-07-26T07:18:06.708 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6261: TEST_request_scrub_priority: grep -q 1.1 2021-07-26T07:18:06.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T07:18:06.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:18:06.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:18:06.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:18:06.710 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:18:06.711 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:18:06.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:18:06.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:18:06.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:18:06.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:18:06.816 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:18:06.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:18:06.817 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:18:06.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:18:06.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:18:06.819 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:18:06.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:18:06.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:18:06.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:18:06.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:18:06.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:18:06.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:18:06.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:18:06.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:18:06.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:18:06.827 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:18:06.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:18:06.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:18:06.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:18:06.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T07:18:06.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T07:18:06.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T07:18:06.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T07:18:06.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:18:06.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:18:06.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:18:06.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:18:06.832 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:18:06.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:18:06.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:18:06.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:18:06.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:18:06.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:18:06.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:18:06.836 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:18:06.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:18:06.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:18:06.836 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:18:06.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:18:06.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:18:06.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:18:06.837 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:18:06.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:18:06.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:18:06.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:18:06.839 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:18:06.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:18:06.840 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:18:06.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:18:06.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:18:06.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:18:06.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T07:18:06.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:18:06.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:18:06.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:18:06.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T07:18:06.845 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:18:06.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:18:06.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:18:06.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T07:18:06.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_scrub_warning td/osd-scrub-repair 2021-07-26T07:18:06.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5859: TEST_scrub_warning: local dir=td/osd-scrub-repair 2021-07-26T07:18:06.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: local poolname=psr_pool 2021-07-26T07:18:06.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: local objname=POBJ 2021-07-26T07:18:06.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: local scrubs=5 2021-07-26T07:18:06.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: local deep_scrubs=5 2021-07-26T07:18:06.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: local i1_day=86400 2021-07-26T07:18:06.848 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: calc 86400 '*' 7 2021-07-26T07:18:06.848 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: calc: awk 'BEGIN{print 86400 * 7}' 2021-07-26T07:18:06.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: local i7_days=604800 2021-07-26T07:18:06.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: calc 86400 '*' 14 2021-07-26T07:18:06.852 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: calc: awk 'BEGIN{print 86400 * 14}' 2021-07-26T07:18:06.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: local i14_days=1209600 2021-07-26T07:18:06.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5867: TEST_scrub_warning: local overdue=0.5 2021-07-26T07:18:06.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5868: TEST_scrub_warning: calc 604800 + 86400 + '(' 604800 '*' 0.5 ')' 2021-07-26T07:18:06.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: calc: awk 'BEGIN{print 604800 + 86400 + ( 604800 * 0.5 )}' 2021-07-26T07:18:06.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5868: TEST_scrub_warning: local conf_overdue_seconds=993600 2021-07-26T07:18:06.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: calc 1209600 + 86400 + '(' 1209600 '*' 0.5 ')' 2021-07-26T07:18:06.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: calc: awk 'BEGIN{print 1209600 + 86400 + ( 1209600 * 0.5 )}' 2021-07-26T07:18:06.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: local pool_overdue_seconds=1900800 2021-07-26T07:18:06.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5871: TEST_scrub_warning: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-07-26T07:18:06.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T07:18:06.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:18:06.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:18:06.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:18:06.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T07:18:06.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-07-26T07:18:07.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:18:07.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:18:07.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:18:07.227 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:18:07.227 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:18:07.227 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:18:07.227 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:18:07.228 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-07-26T07:18:07.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:18:07.461 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:18:07.461 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:18:07.461 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:18:07.461 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:18:07.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:18:07.462 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:18:07.462 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:18:07.462 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:18:07.463 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:18:07.463 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:18:07.463 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:18:07.463 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:18:07.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:18:07.464 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T07:18:07.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:18:07.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:18:07.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:18:07.558 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:18:07.558 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:18:07.558 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:18:07.558 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:18:07.559 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:18:07.559 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:18:07.559 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:18:07.559 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:18:07.560 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:18:07.560 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:18:07.560 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T07:18:07.653 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5872: TEST_scrub_warning: run_mgr td/osd-scrub-repair x --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2021-07-26T07:18:07.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T07:18:07.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:18:07.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:18:07.654 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:18:07.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T07:18:07.655 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:18:08.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:18:08.488 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:18:08.488 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:18:08.488 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:18:08.488 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:18:08.489 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:18:08.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:18:08.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:18:08.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2021-07-26T07:18:08.516 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5873: TEST_scrub_warning: run_osd td/osd-scrub-repair 0 --osd_scrub_backoff_ratio=0 2021-07-26T07:18:08.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:18:08.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:18:08.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:18:08.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:18:08.517 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:18:08.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:18:08.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:18:08.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:18:08.518 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:18:08.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:18:08.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:18:08.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:18:08.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:18:08.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:18:08.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:18:08.520 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:18:08.520 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:18:08.520 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:18:08.520 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:18:08.521 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:18:08.521 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:18:08.521 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:18:08.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:18:08.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:18:08.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:18:08.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:18:08.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:18:08.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:18:08.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:18:08.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:18:08.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:18:08.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:18:08.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_backoff_ratio=0 2021-07-26T07:18:08.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:18:08.525 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:18:08.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5cb946d4-da7c-41b8-b7a1-596032faf097 2021-07-26T07:18:08.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5cb946d4-da7c-41b8-b7a1-596032faf097' 2021-07-26T07:18:08.526 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 5cb946d4-da7c-41b8-b7a1-596032faf097 2021-07-26T07:18:08.527 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:18:08.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCwYf5gnhCgHxAAkFIoWc4ap0fNtzmlOWlByA== 2021-07-26T07:18:08.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCwYf5gnhCgHxAAkFIoWc4ap0fNtzmlOWlByA=="}' 2021-07-26T07:18:08.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5cb946d4-da7c-41b8-b7a1-596032faf097 -i td/osd-scrub-repair/0/new.json 2021-07-26T07:18:08.835 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:18:08.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T07:18:08.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_backoff_ratio=0 --mkfs --key AQCwYf5gnhCgHxAAkFIoWc4ap0fNtzmlOWlByA== --osd-uuid 5cb946d4-da7c-41b8-b7a1-596032faf097 2021-07-26T07:18:08.871 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:18:08.870+0000 7fa560f360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:08.872 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:18:08.872+0000 7fa560f360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:08.873 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:18:08.873+0000 7fa560f360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:08.874 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:18:08.873+0000 7fa560f360c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:18:08.874 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:18:08.873+0000 7fa560f360c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:18:08.875 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:18:08.873+0000 7fa560f360c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:18:08.952 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:18:08.951+0000 7fa560f360c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T07:18:14.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T07:18:14.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:18:14.612 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:18:14.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:18:14.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:18:14.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:18:14.954 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:18:14.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:18:14.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_backoff_ratio=0 2021-07-26T07:18:14.955 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:18:14.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:18:14.976 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:18:14.975+0000 7f4aba8e50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:14.977 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:18:14.977+0000 7f4aba8e50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:14.978 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:18:14.978+0000 7f4aba8e50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:15.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:18:15.289 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:18:15.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:18:15.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:18:15.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:18:15.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:15.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:18:15.291 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:18:15.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:15.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:18:15.497 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:18:15.497+0000 7f4aba8e50c0 -1 Falling back to public interface 2021-07-26T07:18:15.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:18:16.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:18:16.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:16.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:18:16.562 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:18:16.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:16.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:18:16.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:18:17.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:18:17.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:17.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:18:17.839 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:18:17.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:17.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:18:17.908 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:18:17.907+0000 7f4aba8e50c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:18:18.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:18:19.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:18:19.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:19.151 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:18:19.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:18:19.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:19.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:18:19.443 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/439676,v1:127.0.0.1:6803/439676] [v2:127.0.0.1:6804/439676,v1:127.0.0.1:6805/439676] exists,up 5cb946d4-da7c-41b8-b7a1-596032faf097 2021-07-26T07:18:19.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:18:19.445 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:18:19.446 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:18:19.446 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: expr 5 + 5 2021-07-26T07:18:19.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: seq 1 10 2021-07-26T07:18:19.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:18:19.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: create_pool psr_pool-1 1 1 2021-07-26T07:18:19.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-1 1 1 2021-07-26T07:18:19.905 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'psr_pool-1' created 2021-07-26T07:18:19.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:18:20.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: wait_for_clean 2021-07-26T07:18:20.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:18:20.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:18:20.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:18:20.917 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:18:20.918 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:18:20.919 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:18:20.919 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:18:20.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:18:20.919 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:18:20.920 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:18:21.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:18:21.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:18:21.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:18:21.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:18:21.029 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:18:21.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:18:21.317 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:18:21.318 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:21.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:18:21.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836482 2021-07-26T07:18:21.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836482 2021-07-26T07:18:21.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836482' 2021-07-26T07:18:21.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:21.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836482 2021-07-26T07:18:21.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:21.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:18:21.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836482 2021-07-26T07:18:21.438 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:21.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836482 2021-07-26T07:18:21.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836482' 2021-07-26T07:18:21.439 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836482 2021-07-26T07:18:21.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:21.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836481 -lt 21474836482 2021-07-26T07:18:21.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:18:22.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:18:22.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:22.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836481 -lt 21474836482 2021-07-26T07:18:22.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:18:23.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:18:23.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:24.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836482 -lt 21474836482 2021-07-26T07:18:24.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:18:24.245 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:24.245 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:24.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:18:24.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:18:24.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:18:24.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:18:24.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:18:24.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:18:24.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:18:24.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:18:24.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:18:24.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:18:24.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:24.908 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:25.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:18:25.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:18:25.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:18:25.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: '[' 1 = 1 ']' 2021-07-26T07:18:25.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: ceph osd pool set psr_pool-1 scrub_max_interval 1209600 2021-07-26T07:18:25.790 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 1 scrub_max_interval to 1209600 2021-07-26T07:18:25.800 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:18:25.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 1 = 6 ']' 2021-07-26T07:18:25.801 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:18:25.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: create_pool psr_pool-2 1 1 2021-07-26T07:18:25.802 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-2 1 1 2021-07-26T07:18:26.283 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'psr_pool-2' created 2021-07-26T07:18:26.292 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:18:27.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: wait_for_clean 2021-07-26T07:18:27.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:18:27.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:18:27.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:18:27.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:18:27.296 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:18:27.297 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:18:27.298 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:18:27.298 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:18:27.298 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:18:27.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:18:27.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:18:27.406 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:18:27.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:18:27.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:18:27.407 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:18:27.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:18:27.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:18:27.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:27.707 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:18:27.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836484 2021-07-26T07:18:27.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836484 2021-07-26T07:18:27.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484' 2021-07-26T07:18:27.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:27.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836484 2021-07-26T07:18:27.828 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:27.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:18:27.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836484 2021-07-26T07:18:27.830 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:27.831 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836484 2021-07-26T07:18:27.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2021-07-26T07:18:27.833 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836484 2021-07-26T07:18:27.834 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:28.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836483 -lt 21474836484 2021-07-26T07:18:28.093 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:18:29.094 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:18:29.094 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:29.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836484 -lt 21474836484 2021-07-26T07:18:29.369 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:18:29.370 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:29.370 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:29.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 2 == 0 2021-07-26T07:18:29.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:18:29.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:18:29.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:18:29.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:18:29.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:18:29.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:18:29.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:18:30.041 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:18:30.042 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:18:30.042 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:30.042 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:30.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 2 2021-07-26T07:18:30.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:18:30.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:18:30.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: '[' 2 = 1 ']' 2021-07-26T07:18:30.369 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:18:30.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 2 = 6 ']' 2021-07-26T07:18:30.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:18:30.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: create_pool psr_pool-3 1 1 2021-07-26T07:18:30.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-3 1 1 2021-07-26T07:18:30.893 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'psr_pool-3' created 2021-07-26T07:18:30.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:18:31.904 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: wait_for_clean 2021-07-26T07:18:31.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:18:31.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:18:31.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:18:31.905 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:18:31.905 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:18:31.906 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:18:31.907 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:18:31.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:18:31.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:18:31.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:18:32.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:18:32.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:18:32.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:18:32.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:18:32.013 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:18:32.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:18:32.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:18:32.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:32.281 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:18:32.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836486 2021-07-26T07:18:32.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836486 2021-07-26T07:18:32.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836486' 2021-07-26T07:18:32.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:32.440 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836486 2021-07-26T07:18:32.440 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:32.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:18:32.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836486 2021-07-26T07:18:32.442 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:32.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836486 2021-07-26T07:18:32.443 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2021-07-26T07:18:32.443 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836486 2021-07-26T07:18:32.444 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:32.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836485 -lt 21474836486 2021-07-26T07:18:32.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:18:33.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:18:33.710 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:33.987 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836486 2021-07-26T07:18:33.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:18:33.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:33.988 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:34.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 3 == 0 2021-07-26T07:18:34.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:18:34.313 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:18:34.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:18:34.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:18:34.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:18:34.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:18:34.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:18:34.572 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=3 2021-07-26T07:18:34.572 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:18:34.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:34.573 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:34.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 3 = 3 2021-07-26T07:18:34.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:18:34.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:18:34.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: '[' 3 = 1 ']' 2021-07-26T07:18:34.898 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:18:34.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 3 = 6 ']' 2021-07-26T07:18:34.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:18:34.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: create_pool psr_pool-4 1 1 2021-07-26T07:18:34.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-4 1 1 2021-07-26T07:18:35.334 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'psr_pool-4' created 2021-07-26T07:18:35.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:18:36.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: wait_for_clean 2021-07-26T07:18:36.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:18:36.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:18:36.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:18:36.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:18:36.348 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:18:36.349 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:18:36.349 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:18:36.349 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:18:36.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:18:36.350 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:18:36.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:18:36.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:18:36.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:18:36.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:18:36.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:18:36.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:18:36.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:18:36.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:36.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:18:36.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836488 2021-07-26T07:18:36.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836488 2021-07-26T07:18:36.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488' 2021-07-26T07:18:36.844 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:36.845 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836488 2021-07-26T07:18:36.846 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:36.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:18:36.847 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836488 2021-07-26T07:18:36.847 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:36.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836488 2021-07-26T07:18:36.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-07-26T07:18:36.848 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836488 2021-07-26T07:18:36.849 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:37.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836488 2021-07-26T07:18:37.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:18:38.114 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:18:38.115 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:38.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836488 -lt 21474836488 2021-07-26T07:18:38.386 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:18:38.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:38.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:38.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T07:18:38.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:18:38.711 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:18:38.711 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:18:38.712 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:18:38.712 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:18:38.713 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:18:38.713 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:18:39.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T07:18:39.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:18:39.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:39.013 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:39.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T07:18:39.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:18:39.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:18:39.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: '[' 4 = 1 ']' 2021-07-26T07:18:39.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:18:39.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 4 = 6 ']' 2021-07-26T07:18:39.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:18:39.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: create_pool psr_pool-5 1 1 2021-07-26T07:18:39.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-5 1 1 2021-07-26T07:18:39.843 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'psr_pool-5' created 2021-07-26T07:18:39.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:18:40.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: wait_for_clean 2021-07-26T07:18:40.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:18:40.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:18:40.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:18:40.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:18:40.856 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:18:40.857 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:18:40.857 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:18:40.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:18:40.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:18:40.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:18:40.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:18:40.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:18:40.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:18:40.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:18:40.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:18:41.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:18:41.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:18:41.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:41.234 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:18:41.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836490 2021-07-26T07:18:41.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836490 2021-07-26T07:18:41.365 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836490' 2021-07-26T07:18:41.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:41.366 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836490 2021-07-26T07:18:41.367 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:41.368 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:18:41.368 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836490 2021-07-26T07:18:41.369 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:41.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836490 2021-07-26T07:18:41.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-07-26T07:18:41.369 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836490 2021-07-26T07:18:41.370 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:41.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836490 2021-07-26T07:18:41.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:18:42.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:18:42.676 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:42.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836490 2021-07-26T07:18:42.954 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:18:43.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:18:43.953 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:44.227 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836490 -lt 21474836490 2021-07-26T07:18:44.228 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:18:44.229 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:44.229 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:44.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 5 == 0 2021-07-26T07:18:44.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:18:44.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:18:44.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:18:44.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:18:44.594 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:18:44.595 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:18:44.595 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:18:44.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=5 2021-07-26T07:18:44.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:18:44.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:44.896 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:45.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 5 = 5 2021-07-26T07:18:45.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:18:45.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:18:45.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: '[' 5 = 1 ']' 2021-07-26T07:18:45.220 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:18:45.221 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 5 = 6 ']' 2021-07-26T07:18:45.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:18:45.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: create_pool psr_pool-6 1 1 2021-07-26T07:18:45.222 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-6 1 1 2021-07-26T07:18:45.628 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'psr_pool-6' created 2021-07-26T07:18:45.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:18:46.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: wait_for_clean 2021-07-26T07:18:46.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:18:46.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:18:46.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:18:46.640 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:18:46.641 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:18:46.641 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:18:46.642 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:18:46.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:18:46.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:18:46.642 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:18:46.744 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:18:46.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:18:46.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:18:46.745 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:18:46.746 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:18:47.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:18:47.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:18:47.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:47.010 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:18:47.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836492 2021-07-26T07:18:47.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836492 2021-07-26T07:18:47.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836492' 2021-07-26T07:18:47.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:47.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836492 2021-07-26T07:18:47.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:47.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:18:47.170 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836492 2021-07-26T07:18:47.171 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:47.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836492 2021-07-26T07:18:47.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2021-07-26T07:18:47.172 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836492 2021-07-26T07:18:47.173 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:47.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836491 -lt 21474836492 2021-07-26T07:18:47.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:18:48.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:18:48.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:48.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836491 -lt 21474836492 2021-07-26T07:18:48.713 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:18:49.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:18:49.715 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:49.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836492 -lt 21474836492 2021-07-26T07:18:49.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:18:49.993 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:49.994 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:50.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 6 == 0 2021-07-26T07:18:50.362 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:18:50.362 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:18:50.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:18:50.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:18:50.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:18:50.363 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:18:50.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:18:50.623 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=6 2021-07-26T07:18:50.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:18:50.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:50.624 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:50.947 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 6 = 6 2021-07-26T07:18:50.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:18:50.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:18:50.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: '[' 6 = 1 ']' 2021-07-26T07:18:50.948 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:18:50.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 6 = 6 ']' 2021-07-26T07:18:50.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: ceph osd pool set psr_pool-6 deep_scrub_interval 1209600 2021-07-26T07:18:51.512 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 6 deep_scrub_interval to 1209600 2021-07-26T07:18:51.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:18:51.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: create_pool psr_pool-7 1 1 2021-07-26T07:18:51.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-7 1 1 2021-07-26T07:18:51.923 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'psr_pool-7' created 2021-07-26T07:18:51.932 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:18:52.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: wait_for_clean 2021-07-26T07:18:52.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:18:52.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:18:52.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:18:52.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:18:52.935 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:18:52.936 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:18:52.936 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:18:52.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:18:52.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:18:52.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:18:53.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:18:53.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:18:53.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:18:53.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:18:53.047 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:18:53.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:18:53.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:18:53.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:53.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:18:53.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836494 2021-07-26T07:18:53.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836494 2021-07-26T07:18:53.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494' 2021-07-26T07:18:53.434 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:53.434 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836494 2021-07-26T07:18:53.435 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:53.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:18:53.438 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836494 2021-07-26T07:18:53.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:53.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836494 2021-07-26T07:18:53.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2021-07-26T07:18:53.441 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836494 2021-07-26T07:18:53.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:53.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836493 -lt 21474836494 2021-07-26T07:18:53.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:18:54.704 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:18:54.705 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:54.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836493 -lt 21474836494 2021-07-26T07:18:54.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:18:55.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:18:55.983 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:56.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836495 -lt 21474836494 2021-07-26T07:18:56.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:18:56.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:56.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:56.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 7 == 0 2021-07-26T07:18:56.590 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:18:56.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:18:56.590 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:18:56.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:18:56.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:18:56.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:18:56.591 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:18:56.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=7 2021-07-26T07:18:56.893 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:18:56.893 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:56.893 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:57.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 7 = 7 2021-07-26T07:18:57.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:18:57.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:18:57.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: '[' 7 = 1 ']' 2021-07-26T07:18:57.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:18:57.219 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 7 = 6 ']' 2021-07-26T07:18:57.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:18:57.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: create_pool psr_pool-8 1 1 2021-07-26T07:18:57.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-8 1 1 2021-07-26T07:18:57.634 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'psr_pool-8' created 2021-07-26T07:18:57.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:18:58.645 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: wait_for_clean 2021-07-26T07:18:58.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:18:58.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:18:58.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:18:58.646 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:18:58.647 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:18:58.647 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:18:58.648 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:18:58.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:18:58.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:18:58.648 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:18:58.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:18:58.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:18:58.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:18:58.755 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:18:58.755 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:18:59.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:18:59.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:18:59.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:59.022 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:18:59.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836496 2021-07-26T07:18:59.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836496 2021-07-26T07:18:59.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496' 2021-07-26T07:18:59.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:59.142 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836496 2021-07-26T07:18:59.142 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:59.142 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:18:59.143 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836496 2021-07-26T07:18:59.143 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:59.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836496 2021-07-26T07:18:59.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-07-26T07:18:59.145 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836496 2021-07-26T07:18:59.145 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:59.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836495 -lt 21474836496 2021-07-26T07:18:59.411 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:19:00.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:19:00.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:19:00.689 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836495 -lt 21474836496 2021-07-26T07:19:00.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:19:01.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T07:19:01.691 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:19:01.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836497 -lt 21474836496 2021-07-26T07:19:01.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:19:01.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:01.970 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:02.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 8 == 0 2021-07-26T07:19:02.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:02.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:02.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:02.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:02.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:02.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:02.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:02.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=8 2021-07-26T07:19:02.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:02.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:02.596 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:02.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 8 = 8 2021-07-26T07:19:02.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:19:02.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:19:02.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: '[' 8 = 1 ']' 2021-07-26T07:19:02.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:19:02.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 8 = 6 ']' 2021-07-26T07:19:02.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:19:02.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: create_pool psr_pool-9 1 1 2021-07-26T07:19:02.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-9 1 1 2021-07-26T07:19:03.427 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'psr_pool-9' created 2021-07-26T07:19:03.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:19:04.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: wait_for_clean 2021-07-26T07:19:04.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:19:04.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:19:04.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:19:04.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:19:04.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:19:04.440 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:19:04.440 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:19:04.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:19:04.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:19:04.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:19:04.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:19:04.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:19:04.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:19:04.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:19:04.551 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:19:04.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:19:04.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:19:04.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:19:04.816 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:19:04.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836499 2021-07-26T07:19:04.973 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836499 2021-07-26T07:19:04.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836499' 2021-07-26T07:19:04.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:19:04.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836499 2021-07-26T07:19:04.975 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:19:04.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:19:04.976 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836499 2021-07-26T07:19:04.976 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:19:04.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836499 2021-07-26T07:19:04.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2021-07-26T07:19:04.977 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836499 2021-07-26T07:19:04.978 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:19:05.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836499 -lt 21474836499 2021-07-26T07:19:05.245 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:19:05.246 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:05.246 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:05.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 9 == 0 2021-07-26T07:19:05.573 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:05.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:05.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:05.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:05.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:05.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:05.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:05.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=9 2021-07-26T07:19:05.837 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:05.837 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:05.837 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:06.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 9 = 9 2021-07-26T07:19:06.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:19:06.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:19:06.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: '[' 9 = 1 ']' 2021-07-26T07:19:06.161 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:19:06.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 9 = 6 ']' 2021-07-26T07:19:06.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:19:06.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: create_pool psr_pool-10 1 1 2021-07-26T07:19:06.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-10 1 1 2021-07-26T07:19:06.592 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'psr_pool-10' created 2021-07-26T07:19:06.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:19:07.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: wait_for_clean 2021-07-26T07:19:07.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:19:07.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:19:07.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:19:07.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:19:07.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:19:07.608 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:19:07.608 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:19:07.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:19:07.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:19:07.609 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:19:07.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:19:07.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:19:07.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:19:07.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:19:07.716 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:19:07.980 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:19:07.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:19:07.981 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:19:07.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:19:08.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836500 2021-07-26T07:19:08.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836500 2021-07-26T07:19:08.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836500' 2021-07-26T07:19:08.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:19:08.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836500 2021-07-26T07:19:08.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:19:08.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:19:08.143 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836500 2021-07-26T07:19:08.143 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:19:08.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836500 2021-07-26T07:19:08.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2021-07-26T07:19:08.144 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836500 2021-07-26T07:19:08.144 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:19:08.407 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836499 -lt 21474836500 2021-07-26T07:19:08.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:19:09.408 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:19:09.409 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:19:09.687 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836500 -lt 21474836500 2021-07-26T07:19:09.688 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:19:09.688 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:09.688 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:10.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 10 == 0 2021-07-26T07:19:10.049 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:10.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:10.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:10.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:10.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:10.050 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:10.051 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:10.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=10 2021-07-26T07:19:10.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:10.352 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:10.353 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:10.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 10 = 10 2021-07-26T07:19:10.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:19:10.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:19:10.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: '[' 10 = 1 ']' 2021-07-26T07:19:10.678 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:19:10.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 10 = 6 ']' 2021-07-26T07:19:10.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: TEST_scrub_warning: local primary=0 2021-07-26T07:19:10.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: TEST_scrub_warning: ceph osd set noscrub 2021-07-26T07:19:11.067 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:19:11.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5893: TEST_scrub_warning: ceph osd set nodeep-scrub 2021-07-26T07:19:11.594 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:19:11.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph config set global osd_scrub_interval_randomize_ratio 0 2021-07-26T07:19:11.982 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5895: TEST_scrub_warning: ceph config set global osd_deep_scrub_randomize_ratio 0 2021-07-26T07:19:12.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: ceph config set global osd_scrub_max_interval 604800 2021-07-26T07:19:12.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: ceph config set global osd_deep_scrub_interval 604800 2021-07-26T07:19:13.010 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: seq 1 5 2021-07-26T07:19:13.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2021-07-26T07:19:13.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: '[' 1 = 1 ']' 2021-07-26T07:19:13.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5904: TEST_scrub_warning: overdue_seconds=1900800 2021-07-26T07:19:13.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: expr 1900800 + 100 2021-07-26T07:19:13.012 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: ceph tell 1.0 scrub 1900900 2021-07-26T07:19:13.143 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:19:13.143 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:19:13.144 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-04T07:15:53.141421+0000" 2021-07-26T07:19:13.144 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:19:13.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2021-07-26T07:19:13.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: '[' 2 = 1 ']' 2021-07-26T07:19:13.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5906: TEST_scrub_warning: overdue_seconds=993600 2021-07-26T07:19:13.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: expr 993600 + 200 2021-07-26T07:19:13.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: ceph tell 2.0 scrub 993800 2021-07-26T07:19:13.292 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:19:13.292 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:19:13.292 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-14T19:14:13.291773+0000" 2021-07-26T07:19:13.292 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:19:13.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2021-07-26T07:19:13.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: '[' 3 = 1 ']' 2021-07-26T07:19:13.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5906: TEST_scrub_warning: overdue_seconds=993600 2021-07-26T07:19:13.303 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: expr 993600 + 300 2021-07-26T07:19:13.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: ceph tell 3.0 scrub 993900 2021-07-26T07:19:13.416 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:19:13.417 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:19:13.417 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-14T19:12:33.416524+0000" 2021-07-26T07:19:13.417 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:19:13.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2021-07-26T07:19:13.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: '[' 4 = 1 ']' 2021-07-26T07:19:13.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5906: TEST_scrub_warning: overdue_seconds=993600 2021-07-26T07:19:13.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: expr 993600 + 400 2021-07-26T07:19:13.428 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: ceph tell 4.0 scrub 994000 2021-07-26T07:19:13.538 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:19:13.539 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:19:13.539 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-14T19:10:53.538689+0000" 2021-07-26T07:19:13.539 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:19:13.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2021-07-26T07:19:13.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: '[' 5 = 1 ']' 2021-07-26T07:19:13.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5906: TEST_scrub_warning: overdue_seconds=993600 2021-07-26T07:19:13.549 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: expr 993600 + 500 2021-07-26T07:19:13.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: ceph tell 5.0 scrub 994100 2021-07-26T07:19:13.686 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:19:13.687 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": false, 2021-07-26T07:19:13.687 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-14T19:09:13.685807+0000" 2021-07-26T07:19:13.687 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:19:13.696 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:19:13.698 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: expr 5 + 5 2021-07-26T07:19:13.698 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: seq 6 10 2021-07-26T07:19:13.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:19:13.700 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5913: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:19:13.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5913: TEST_scrub_warning: '[' 6 = 6 ']' 2021-07-26T07:19:13.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5915: TEST_scrub_warning: overdue_seconds=1900800 2021-07-26T07:19:13.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: expr 1900800 + 600 2021-07-26T07:19:13.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: ceph tell 6.0 deep_scrub 1901400 2021-07-26T07:19:13.809 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:19:13.810 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": true, 2021-07-26T07:19:13.810 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-04T07:07:33.809511+0000" 2021-07-26T07:19:13.810 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:19:13.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:19:13.820 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5913: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:19:13.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5913: TEST_scrub_warning: '[' 7 = 6 ']' 2021-07-26T07:19:13.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: overdue_seconds=993600 2021-07-26T07:19:13.821 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: expr 993600 + 700 2021-07-26T07:19:13.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: ceph tell 7.0 deep_scrub 994300 2021-07-26T07:19:13.929 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:19:13.929 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": true, 2021-07-26T07:19:13.929 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-14T19:05:53.929143+0000" 2021-07-26T07:19:13.930 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:19:13.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:19:13.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5913: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:19:13.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5913: TEST_scrub_warning: '[' 8 = 6 ']' 2021-07-26T07:19:13.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: overdue_seconds=993600 2021-07-26T07:19:13.941 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: expr 993600 + 800 2021-07-26T07:19:13.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: ceph tell 8.0 deep_scrub 994400 2021-07-26T07:19:14.049 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:19:14.049 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": true, 2021-07-26T07:19:14.050 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-14T19:04:14.049356+0000" 2021-07-26T07:19:14.050 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:19:14.059 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:19:14.060 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5913: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:19:14.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5913: TEST_scrub_warning: '[' 9 = 6 ']' 2021-07-26T07:19:14.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: overdue_seconds=993600 2021-07-26T07:19:14.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: expr 993600 + 900 2021-07-26T07:19:14.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: ceph tell 9.0 deep_scrub 994500 2021-07-26T07:19:14.169 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:19:14.169 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": true, 2021-07-26T07:19:14.170 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-14T19:02:34.169281+0000" 2021-07-26T07:19:14.170 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:19:14.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2021-07-26T07:19:14.179 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5913: TEST_scrub_warning: expr 5 + 1 2021-07-26T07:19:14.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5913: TEST_scrub_warning: '[' 10 = 6 ']' 2021-07-26T07:19:14.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: overdue_seconds=993600 2021-07-26T07:19:14.181 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: expr 993600 + 1000 2021-07-26T07:19:14.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: ceph tell 10.0 deep_scrub 994600 2021-07-26T07:19:14.289 INFO:tasks.workunit.client.0.smithi071.stdout:{ 2021-07-26T07:19:14.290 INFO:tasks.workunit.client.0.smithi071.stdout: "deep": true, 2021-07-26T07:19:14.290 INFO:tasks.workunit.client.0.smithi071.stdout: "stamp": "2021-07-14T19:00:54.289821+0000" 2021-07-26T07:19:14.290 INFO:tasks.workunit.client.0.smithi071.stdout:} 2021-07-26T07:19:14.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5921: TEST_scrub_warning: flush_pg_stats 2021-07-26T07:19:14.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:19:14.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:19:14.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:19:14.565 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:19:14.566 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:19:14.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:19:14.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836503 2021-07-26T07:19:14.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836503 2021-07-26T07:19:14.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836503' 2021-07-26T07:19:14.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:19:14.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836503 2021-07-26T07:19:14.728 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:19:14.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:19:14.729 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836503 2021-07-26T07:19:14.730 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:19:14.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836503 2021-07-26T07:19:14.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2021-07-26T07:19:14.731 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836503 2021-07-26T07:19:14.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:19:14.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836501 -lt 21474836503 2021-07-26T07:19:14.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:19:15.995 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:19:15.996 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:19:16.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836503 -lt 21474836503 2021-07-26T07:19:16.266 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: ceph health 2021-07-26T07:19:16.581 INFO:tasks.workunit.client.0.smithi071.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 5 pgs not scrubbed in time; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2021-07-26T07:19:16.592 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5924: TEST_scrub_warning: ceph health detail 2021-07-26T07:19:16.945 INFO:tasks.workunit.client.0.smithi071.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 5 pgs not scrubbed in time; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2021-07-26T07:19:16.946 INFO:tasks.workunit.client.0.smithi071.stdout:[WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2021-07-26T07:19:16.946 INFO:tasks.workunit.client.0.smithi071.stdout:[WRN] PG_NOT_DEEP_SCRUBBED: 5 pgs not deep-scrubbed in time 2021-07-26T07:19:16.946 INFO:tasks.workunit.client.0.smithi071.stdout: pg 10.0 not deep-scrubbed since 2021-07-14T19:00:54.289821+0000 2021-07-26T07:19:16.946 INFO:tasks.workunit.client.0.smithi071.stdout: pg 9.0 not deep-scrubbed since 2021-07-14T19:02:34.169281+0000 2021-07-26T07:19:16.946 INFO:tasks.workunit.client.0.smithi071.stdout: pg 8.0 not deep-scrubbed since 2021-07-14T19:04:14.049356+0000 2021-07-26T07:19:16.947 INFO:tasks.workunit.client.0.smithi071.stdout: pg 7.0 not deep-scrubbed since 2021-07-14T19:05:53.929143+0000 2021-07-26T07:19:16.947 INFO:tasks.workunit.client.0.smithi071.stdout: pg 6.0 not deep-scrubbed since 2021-07-04T07:07:33.809511+0000 2021-07-26T07:19:16.947 INFO:tasks.workunit.client.0.smithi071.stdout:[WRN] PG_NOT_SCRUBBED: 5 pgs not scrubbed in time 2021-07-26T07:19:16.947 INFO:tasks.workunit.client.0.smithi071.stdout: pg 2.0 not scrubbed since 2021-07-14T19:14:13.291773+0000 2021-07-26T07:19:16.948 INFO:tasks.workunit.client.0.smithi071.stdout: pg 1.0 not scrubbed since 2021-07-04T07:15:53.141421+0000 2021-07-26T07:19:16.948 INFO:tasks.workunit.client.0.smithi071.stdout: pg 3.0 not scrubbed since 2021-07-14T19:12:33.416524+0000 2021-07-26T07:19:16.948 INFO:tasks.workunit.client.0.smithi071.stdout: pg 4.0 not scrubbed since 2021-07-14T19:10:53.538689+0000 2021-07-26T07:19:16.948 INFO:tasks.workunit.client.0.smithi071.stdout: pg 5.0 not scrubbed since 2021-07-14T19:09:13.685807+0000 2021-07-26T07:19:16.948 INFO:tasks.workunit.client.0.smithi071.stdout:[WRN] POOL_NO_REDUNDANCY: 10 pool(s) have no replicas configured 2021-07-26T07:19:16.949 INFO:tasks.workunit.client.0.smithi071.stdout: pool 'psr_pool-1' has no replicas configured 2021-07-26T07:19:16.949 INFO:tasks.workunit.client.0.smithi071.stdout: pool 'psr_pool-2' has no replicas configured 2021-07-26T07:19:16.949 INFO:tasks.workunit.client.0.smithi071.stdout: pool 'psr_pool-3' has no replicas configured 2021-07-26T07:19:16.949 INFO:tasks.workunit.client.0.smithi071.stdout: pool 'psr_pool-4' has no replicas configured 2021-07-26T07:19:16.949 INFO:tasks.workunit.client.0.smithi071.stdout: pool 'psr_pool-5' has no replicas configured 2021-07-26T07:19:16.950 INFO:tasks.workunit.client.0.smithi071.stdout: pool 'psr_pool-6' has no replicas configured 2021-07-26T07:19:16.950 INFO:tasks.workunit.client.0.smithi071.stdout: pool 'psr_pool-7' has no replicas configured 2021-07-26T07:19:16.950 INFO:tasks.workunit.client.0.smithi071.stdout: pool 'psr_pool-8' has no replicas configured 2021-07-26T07:19:16.950 INFO:tasks.workunit.client.0.smithi071.stdout: pool 'psr_pool-9' has no replicas configured 2021-07-26T07:19:16.951 INFO:tasks.workunit.client.0.smithi071.stdout: pool 'psr_pool-10' has no replicas configured 2021-07-26T07:19:16.951 INFO:tasks.workunit.client.0.smithi071.stdout:[WRN] TOO_FEW_OSDS: OSD count 1 < osd_pool_default_size 3 2021-07-26T07:19:16.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5925: TEST_scrub_warning: ceph health 2021-07-26T07:19:16.957 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5925: TEST_scrub_warning: grep -q '5 pgs not deep-scrubbed in time' 2021-07-26T07:19:17.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5926: TEST_scrub_warning: ceph health 2021-07-26T07:19:17.320 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5926: TEST_scrub_warning: grep -q '5 pgs not scrubbed in time' 2021-07-26T07:19:17.682 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5927: TEST_scrub_warning: ceph health detail 2021-07-26T07:19:17.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5927: TEST_scrub_warning: grep 'not scrubbed since' 2021-07-26T07:19:17.683 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5927: TEST_scrub_warning: wc -l 2021-07-26T07:19:18.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5927: TEST_scrub_warning: COUNT=5 2021-07-26T07:19:18.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5928: TEST_scrub_warning: '[' 5 '!=' 5 ']' 2021-07-26T07:19:18.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_scrub_warning: ceph health detail 2021-07-26T07:19:18.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_scrub_warning: grep 'not deep-scrubbed since' 2021-07-26T07:19:18.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_scrub_warning: wc -l 2021-07-26T07:19:18.358 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_scrub_warning: COUNT=5 2021-07-26T07:19:18.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5933: TEST_scrub_warning: '[' 5 '!=' 5 ']' 2021-07-26T07:19:18.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T07:19:18.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:19:18.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:19:18.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:19:18.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:19:18.360 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:19:18.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:19:18.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:19:18.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:19:18.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:19:18.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:19:18.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:19:18.466 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:19:18.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:19:18.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:19:18.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:19:18.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:19:18.469 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:19:18.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:19:18.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:19:18.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:19:18.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:19:18.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:19:18.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:19:18.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:19:18.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:19:18.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:19:18.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:19:18.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:19:18.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T07:19:18.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T07:19:18.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T07:19:18.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T07:19:18.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:19:18.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:19:18.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:19:18.482 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:19:18.482 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:19:18.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:19:18.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:19:18.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:19:18.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:19:18.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:19:18.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:19:18.485 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:19:18.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:19:18.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:19:18.486 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:19:18.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:19:18.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:19:18.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:19:18.487 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:19:18.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:19:18.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:19:18.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:19:18.489 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:19:18.490 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:19:18.490 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:19:18.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:19:18.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:19:18.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:19:18.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T07:19:18.492 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:19:18.492 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:19:18.493 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:19:18.493 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T07:19:18.495 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:19:18.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:19:18.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:19:18.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T07:19:18.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_skip_non_repair_during_recovery td/osd-scrub-repair 2021-07-26T07:19:18.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:135: TEST_skip_non_repair_during_recovery: local dir=td/osd-scrub-repair 2021-07-26T07:19:18.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:136: TEST_skip_non_repair_during_recovery: local poolname=rbd 2021-07-26T07:19:18.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:138: TEST_skip_non_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2021-07-26T07:19:18.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T07:19:18.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:19:18.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:19:18.497 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:19:18.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T07:19:18.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2021-07-26T07:19:18.853 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:19:18.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:19:18.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:19:18.854 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:19:18.854 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:19:18.855 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:19:18.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:19:18.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2021-07-26T07:19:19.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:19:19.104 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:19:19.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:19:19.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:19:19.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:19:19.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:19:19.107 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:19:19.107 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:19:19.107 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:19:19.107 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:19:19.108 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:19:19.108 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:19:19.108 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:19:19.108 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:19:19.109 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T07:19:19.202 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:19:19.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:19:19.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:19:19.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:19:19.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:19:19.203 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:19:19.204 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:19:19.204 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:19:19.204 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:19:19.204 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:19:19.205 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:19:19.205 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:19:19.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:19:19.206 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T07:19:19.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:139: TEST_skip_non_repair_during_recovery: run_mgr td/osd-scrub-repair x 2021-07-26T07:19:19.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T07:19:19.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:19:19.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:19:19.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:19:19.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T07:19:19.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:19:20.132 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:19:20.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:19:20.133 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:19:20.133 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:19:20.133 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:19:20.134 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:19:20.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:19:20.134 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:19:20.134 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:19:20.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:140: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2021-07-26T07:19:20.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:19:20.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:19:20.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:19:20.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:19:20.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:19:20.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:19:20.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:19:20.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:19:20.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:19:20.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:19:20.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:19:20.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:19:20.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:19:20.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:19:20.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:19:20.163 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:19:20.164 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:19:20.164 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:19:20.164 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:19:20.164 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:19:20.165 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:19:20.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:19:20.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:19:20.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:19:20.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:19:20.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:19:20.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:19:20.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:19:20.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:19:20.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:19:20.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:19:20.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:19:20.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2021-07-26T07:19:20.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:19:20.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:19:20.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eca7be24-aa0d-47bb-9635-6600ed9d5d9c 2021-07-26T07:19:20.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 eca7be24-aa0d-47bb-9635-6600ed9d5d9c' 2021-07-26T07:19:20.169 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 eca7be24-aa0d-47bb-9635-6600ed9d5d9c 2021-07-26T07:19:20.169 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:19:20.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD4Yf5gE0teChAAdbbiLKfOWBEAcbJhGNhMHQ== 2021-07-26T07:19:20.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD4Yf5gE0teChAAdbbiLKfOWBEAcbJhGNhMHQ=="}' 2021-07-26T07:19:20.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eca7be24-aa0d-47bb-9635-6600ed9d5d9c -i td/osd-scrub-repair/0/new.json 2021-07-26T07:19:20.429 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:19:20.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T07:19:20.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQD4Yf5gE0teChAAdbbiLKfOWBEAcbJhGNhMHQ== --osd-uuid eca7be24-aa0d-47bb-9635-6600ed9d5d9c 2021-07-26T07:19:20.467 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:20.466+0000 7efe8ffee0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:20.468 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:20.468+0000 7efe8ffee0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:20.469 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:20.469+0000 7efe8ffee0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:20.470 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:20.469+0000 7efe8ffee0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:19:20.470 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:20.469+0000 7efe8ffee0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:19:20.471 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:20.469+0000 7efe8ffee0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:19:20.529 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:20.528+0000 7efe8ffee0c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T07:19:26.259 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T07:19:26.260 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:19:26.260 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:19:26.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:19:26.261 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:19:26.598 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:19:26.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:19:26.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:19:26.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2021-07-26T07:19:26.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:19:26.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:19:26.622 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:26.621+0000 7ff8d007f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:26.623 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:26.622+0000 7ff8d007f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:26.624 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:26.623+0000 7ff8d007f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:26.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:19:26.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:19:26.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:19:26.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:19:26.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:19:26.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:19:26.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:19:26.913 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:19:26.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:19:26.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:19:27.147 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:27.147+0000 7ff8d007f0c0 -1 Falling back to public interface 2021-07-26T07:19:27.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:19:28.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:19:28.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:19:28.215 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:19:28.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:19:28.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:19:28.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:19:28.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:19:29.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:19:29.490 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:19:29.491 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:19:29.491 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:19:29.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:19:29.492 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:19:29.552 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:29.551+0000 7ff8d007f0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:19:29.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:19:30.809 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:19:30.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:19:30.810 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:19:30.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:19:30.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:19:30.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:19:31.094 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/445213,v1:127.0.0.1:6803/445213] [v2:127.0.0.1:6804/445213,v1:127.0.0.1:6805/445213] exists,up eca7be24-aa0d-47bb-9635-6600ed9d5d9c 2021-07-26T07:19:31.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:19:31.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:19:31.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:19:31.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:143: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2021-07-26T07:19:31.101 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:19:31.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:19:31.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:19:31.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:19:31.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:19:31.102 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:19:31.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:19:31.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:19:31.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:19:31.103 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:19:31.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:19:31.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:19:31.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:19:31.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:19:31.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:19:31.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:19:31.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:19:31.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:19:31.106 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:19:31.106 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:19:31.106 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:19:31.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:19:31.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:19:31.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:19:31.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:19:31.107 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:19:31.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:19:31.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:19:31.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:19:31.108 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:19:31.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:19:31.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:19:31.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2021-07-26T07:19:31.109 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:19:31.110 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:19:31.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c0a92aa-3a96-4dd8-8b9e-ead2afaa656e 2021-07-26T07:19:31.110 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5c0a92aa-3a96-4dd8-8b9e-ead2afaa656e' 2021-07-26T07:19:31.110 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 5c0a92aa-3a96-4dd8-8b9e-ead2afaa656e 2021-07-26T07:19:31.111 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:19:31.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADYv5gudGyBhAAJtGkOSUn+Mza5YeJiGquCA== 2021-07-26T07:19:31.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADYv5gudGyBhAAJtGkOSUn+Mza5YeJiGquCA=="}' 2021-07-26T07:19:31.113 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c0a92aa-3a96-4dd8-8b9e-ead2afaa656e -i td/osd-scrub-repair/1/new.json 2021-07-26T07:19:31.470 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:19:31.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T07:19:31.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQADYv5gudGyBhAAJtGkOSUn+Mza5YeJiGquCA== --osd-uuid 5c0a92aa-3a96-4dd8-8b9e-ead2afaa656e 2021-07-26T07:19:31.510 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:31.509+0000 7f418a2d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:31.511 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:31.511+0000 7f418a2d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:31.513 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:31.512+0000 7f418a2d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:31.513 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:31.512+0000 7f418a2d80c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:19:31.514 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:31.512+0000 7f418a2d80c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:19:31.514 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:31.512+0000 7f418a2d80c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:19:31.570 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:31.569+0000 7f418a2d80c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T07:19:36.958 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T07:19:36.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:19:36.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:19:36.960 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T07:19:36.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:19:37.320 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T07:19:37.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:19:37.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:19:37.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2021-07-26T07:19:37.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:19:37.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:19:37.343 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:37.342+0000 7fbc0c7550c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:37.345 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:37.343+0000 7fbc0c7550c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:37.346 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:37.344+0000 7fbc0c7550c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:37.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:19:37.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:19:37.631 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:19:37.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:19:37.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:19:37.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:19:37.632 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:19:37.633 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:19:37.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:19:37.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:19:37.861 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:37.860+0000 7fbc0c7550c0 -1 Falling back to public interface 2021-07-26T07:19:37.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:19:38.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:19:38.937 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:19:38.938 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:19:38.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:19:38.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:19:38.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:19:39.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:19:40.170 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:40.168+0000 7fbc0c7550c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:19:40.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:19:40.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:19:40.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:19:40.217 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:19:40.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:19:40.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:19:40.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:19:41.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:19:41.603 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:19:41.604 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:19:41.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:19:41.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:19:41.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:19:41.869 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/446612,v1:127.0.0.1:6811/446612] [v2:127.0.0.1:6812/446612,v1:127.0.0.1:6813/446612] exists,up 5c0a92aa-3a96-4dd8-8b9e-ead2afaa656e 2021-07-26T07:19:41.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:19:41.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:19:41.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:19:41.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:146: TEST_skip_non_repair_during_recovery: create_rbd_pool 2021-07-26T07:19:41.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T07:19:42.167 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' does not exist 2021-07-26T07:19:42.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T07:19:42.176 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T07:19:42.647 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'rbd' created 2021-07-26T07:19:42.650 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:19:43.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T07:19:45.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:147: TEST_skip_non_repair_during_recovery: wait_for_clean 2021-07-26T07:19:45.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:19:45.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:19:45.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:19:45.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:19:45.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:19:45.062 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:19:45.063 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:19:45.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:19:45.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:19:45.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:19:45.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:19:45.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:19:45.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:19:45.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:19:45.172 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:19:45.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:19:45.438 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:19:45.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:19:45.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:19:45.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:19:45.557 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836484 2021-07-26T07:19:45.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836484 2021-07-26T07:19:45.558 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484' 2021-07-26T07:19:45.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:19:45.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:19:45.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672963 2021-07-26T07:19:45.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672963 2021-07-26T07:19:45.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484 1-42949672963' 2021-07-26T07:19:45.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:19:45.677 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836484 2021-07-26T07:19:45.677 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:19:45.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:19:45.678 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836484 2021-07-26T07:19:45.679 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:19:45.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836484 2021-07-26T07:19:45.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2021-07-26T07:19:45.680 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836484 2021-07-26T07:19:45.680 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:19:45.945 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836483 -lt 21474836484 2021-07-26T07:19:45.946 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:19:46.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:19:46.950 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:19:47.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836485 -lt 21474836484 2021-07-26T07:19:47.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:19:47.215 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672963 2021-07-26T07:19:47.216 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:19:47.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:19:47.217 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672963 2021-07-26T07:19:47.217 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:19:47.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672963 2021-07-26T07:19:47.218 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2021-07-26T07:19:47.219 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672963 2021-07-26T07:19:47.219 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:19:47.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672964 -lt 42949672963 2021-07-26T07:19:47.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:19:47.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:47.484 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:47.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T07:19:47.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:47.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:47.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:47.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:47.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:47.856 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:47.856 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:48.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T07:19:48.156 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:48.156 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:48.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:48.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T07:19:48.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:19:48.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:19:48.487 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:149: TEST_skip_non_repair_during_recovery: add_something td/osd-scrub-repair rbd 2021-07-26T07:19:48.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:19:48.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=rbd 2021-07-26T07:19:48.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2021-07-26T07:19:48.488 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:19:48.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:19:48.489 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:19:48.876 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:19:48.885 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:19:49.412 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:19:49.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:19:49.421 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:19:49.422 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T07:19:49.650 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:150: TEST_skip_non_repair_during_recovery: get_not_primary rbd SOMETHING 2021-07-26T07:19:49.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=rbd 2021-07-26T07:19:49.651 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=SOMETHING 2021-07-26T07:19:49.651 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary rbd SOMETHING 2021-07-26T07:19:49.652 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=rbd 2021-07-26T07:19:49.652 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T07:19:49.652 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map rbd SOMETHING 2021-07-26T07:19:49.652 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:19:49.921 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T07:19:49.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map rbd SOMETHING 2021-07-26T07:19:49.922 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T07:19:50.230 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:150: TEST_skip_non_repair_during_recovery: scrub_and_not_schedule td/osd-scrub-repair rbd 0 2021-07-26T07:19:50.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:154: scrub_and_not_schedule: local dir=td/osd-scrub-repair 2021-07-26T07:19:50.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:155: scrub_and_not_schedule: local poolname=rbd 2021-07-26T07:19:50.231 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:156: scrub_and_not_schedule: local osd=0 2021-07-26T07:19:50.232 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:161: scrub_and_not_schedule: get_pg rbd SOMETHING 2021-07-26T07:19:50.232 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=rbd 2021-07-26T07:19:50.232 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T07:19:50.232 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map rbd SOMETHING 2021-07-26T07:19:50.233 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:19:50.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:161: scrub_and_not_schedule: local pg=1.3 2021-07-26T07:19:50.500 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2021-07-26T07:19:50.500 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T07:19:50.500 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:19:50.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:19:50.501 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T07:19:50.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: local last_scrub=2021-07-26T07:19:42.560439+0000 2021-07-26T07:19:50.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: ceph pg scrub 1.3 2021-07-26T07:19:51.068 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 1.3 on osd.1 to scrub 2021-07-26T07:19:51.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i=0 )) 2021-07-26T07:19:51.078 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i < 3 )) 2021-07-26T07:19:51.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2021-07-26T07:19:51.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T07:19:51.079 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:19:51.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:19:51.080 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T07:19:51.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: test 2021-07-26T07:19:42.560439+0000 '>' 2021-07-26T07:19:42.560439+0000 2021-07-26T07:19:51.340 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:172: scrub_and_not_schedule: sleep 1 2021-07-26T07:19:52.341 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i++ )) 2021-07-26T07:19:52.342 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i < 3 )) 2021-07-26T07:19:52.343 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2021-07-26T07:19:52.343 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T07:19:52.343 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:19:52.343 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:19:52.343 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T07:19:52.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: test 2021-07-26T07:19:42.560439+0000 '>' 2021-07-26T07:19:42.560439+0000 2021-07-26T07:19:52.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:172: scrub_and_not_schedule: sleep 1 2021-07-26T07:19:53.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i++ )) 2021-07-26T07:19:53.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i < 3 )) 2021-07-26T07:19:53.615 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2021-07-26T07:19:53.616 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.3 2021-07-26T07:19:53.616 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:19:53.616 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:19:53.616 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2021-07-26T07:19:53.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: test 2021-07-26T07:19:42.560439+0000 '>' 2021-07-26T07:19:42.560439+0000 2021-07-26T07:19:53.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:172: scrub_and_not_schedule: sleep 1 2021-07-26T07:19:54.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i++ )) 2021-07-26T07:19:54.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i < 3 )) 2021-07-26T07:19:54.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:178: scrub_and_not_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2021-07-26T07:19:54.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:19:54.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:19:54.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:19:54.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:19:54.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2021-07-26T07:19:54.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:19:54.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:19:54.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:19:54.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:19:54.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:19:54.892 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:19:54.893 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:19:54.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:19:54.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:19:54.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:19:54.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:19:54.996 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2021-07-26T07:19:54.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:19:54.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:19:54.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:19:54.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:19:54.997 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:19:54.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2021-07-26T07:19:56.591 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2021-07-26T07:19:56.592 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2021-07-26T07:19:56.592 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:19:56.592 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2021-07-26T07:19:56.593 INFO:tasks.workunit.client.0.smithi071.stdout:_ 2021-07-26T07:19:56.593 INFO:tasks.workunit.client.0.smithi071.stdout:snapset 2021-07-26T07:19:56.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:19:56.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:19:56.869 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:19:56.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:19:56.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:19:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:19:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:19:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:19:56.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:19:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:19:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:19:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:19:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:19:56.872 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:19:56.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:19:56.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:19:56.873 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:19:56.873 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:19:56.874 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:19:56.874 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:19:56.874 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:19:56.874 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:19:56.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:19:56.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:19:56.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:19:56.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:19:56.875 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:19:56.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:19:56.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:19:56.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:19:56.876 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:19:56.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:19:56.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:19:56.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:19:56.877 INFO:tasks.workunit.client.0.smithi071.stderr:start osd.0 2021-07-26T07:19:56.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:19:56.878 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:19:56.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:19:56.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:19:56.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:19:56.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:19:56.893 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:56.893+0000 7f031e9480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:56.895 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:56.894+0000 7f031e9480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:56.896 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:56.896+0000 7f031e9480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:19:57.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:19:57.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:19:57.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:19:57.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:19:57.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:19:57.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:19:57.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:19:57.150 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:19:57.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:19:57.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:19:57.413 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:57.412+0000 7f031e9480c0 -1 Falling back to public interface 2021-07-26T07:19:57.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:19:58.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:19:58.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:19:58.460 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:19:58.460 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:19:58.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:19:58.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:19:58.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:19:59.736 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:19:59.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:19:59.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:19:59.737 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:19:59.737 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:19:59.738 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:19:59.853 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:19:59.852+0000 7f031e9480c0 -1 osd.0 22 log_to_monitors {default=true} 2021-07-26T07:20:00.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:01.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:01.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:01.090 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:20:01.090 INFO:tasks.workunit.client.0.smithi071.stderr:3 2021-07-26T07:20:01.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:01.091 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:20:01.369 INFO:tasks.workunit.client.0.smithi071.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/448240,v1:127.0.0.1:6803/448240] [v2:127.0.0.1:6804/448240,v1:127.0.0.1:6805/448240] exists,up eca7be24-aa0d-47bb-9635-6600ed9d5d9c 2021-07-26T07:20:01.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:20:01.369 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:20:01.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:20:01.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:20:01.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:20:01.370 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:20:01.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:20:01.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:20:01.371 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:20:01.371 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:20:01.372 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:20:01.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:20:01.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:20:01.372 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:20:01.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:20:01.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:20:01.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:20:01.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:20:01.467 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:20:01.730 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:20:01.731 INFO:tasks.workunit.client.0.smithi071.stderr:1' 2021-07-26T07:20:01.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:20:01.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:01.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:20:01.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=111669149698 2021-07-26T07:20:01.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 111669149698 2021-07-26T07:20:01.889 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149698' 2021-07-26T07:20:01.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:01.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:20:02.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672968 2021-07-26T07:20:02.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672968 2021-07-26T07:20:02.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149698 1-42949672968' 2021-07-26T07:20:02.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:02.009 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-111669149698 2021-07-26T07:20:02.009 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:02.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:20:02.011 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-111669149698 2021-07-26T07:20:02.011 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:02.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=111669149698 2021-07-26T07:20:02.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2021-07-26T07:20:02.011 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.0 seq 111669149698 2021-07-26T07:20:02.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:20:02.274 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 111669149698 2021-07-26T07:20:02.275 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:20:03.276 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:20:03.276 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:20:03.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149698 -lt 111669149698 2021-07-26T07:20:03.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:03.553 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672968 2021-07-26T07:20:03.553 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:03.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:20:03.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672968 2021-07-26T07:20:03.555 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:03.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672968 2021-07-26T07:20:03.556 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-07-26T07:20:03.556 INFO:tasks.workunit.client.0.smithi071.stderr:waiting osd.1 seq 42949672968 2021-07-26T07:20:03.557 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:20:03.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672968 -lt 42949672968 2021-07-26T07:20:03.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:20:03.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:03.823 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:04.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 4 == 0 2021-07-26T07:20:04.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:20:04.148 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:20:04.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:20:04.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:20:04.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:20:04.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:20:04.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:20:04.410 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=4 2021-07-26T07:20:04.411 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:20:04.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:04.412 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:04.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 4 = 4 2021-07-26T07:20:04.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:20:04.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:20:04.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:179: scrub_and_not_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2021-07-26T07:20:04.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:180: scrub_and_not_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2021-07-26T07:20:04.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T07:20:04.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:20:04.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:20:04.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:20:04.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:20:04.772 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:20:04.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:20:04.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:20:04.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:20:04.877 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:20:04.878 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:20:04.879 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:20:04.879 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:20:04.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:20:04.880 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:20:04.881 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:20:04.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:20:04.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:20:04.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:20:04.883 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:20:04.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:20:04.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:20:04.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:20:04.889 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:20:04.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:04.890 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:04.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:20:04.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:20:04.892 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:20:04.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T07:20:04.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T07:20:04.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T07:20:04.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T07:20:04.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:20:04.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:20:04.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:20:04.894 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:20:04.895 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:20:04.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:20:04.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:20:04.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:20:04.896 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:20:04.897 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:20:04.897 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:20:04.897 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:20:04.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:20:04.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:20:04.898 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:20:04.899 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:20:04.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:20:04.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:20:04.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:20:04.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:20:04.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:20:04.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:20:04.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:20:04.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:04.902 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:04.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:20:04.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:20:04.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:20:04.903 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T07:20:04.906 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:20:04.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:04.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:04.907 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T07:20:04.907 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:20:04.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:20:04.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:20:04.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T07:20:04.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_unfound_erasure_coded_appends td/osd-scrub-repair 2021-07-26T07:20:04.908 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:912: TEST_unfound_erasure_coded_appends: unfound_erasure_coded td/osd-scrub-repair false 2021-07-26T07:20:04.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: local dir=td/osd-scrub-repair 2021-07-26T07:20:04.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: local allow_overwrites=false 2021-07-26T07:20:04.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: local poolname=ecpool 2021-07-26T07:20:04.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: local payload=ABCDEF 2021-07-26T07:20:04.909 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:862: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2021-07-26T07:20:04.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T07:20:04.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:20:04.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:20:04.910 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:20:04.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T07:20:04.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T07:20:05.261 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:20:05.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:05.262 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:05.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:05.262 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:05.263 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:05.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:05.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:20:05.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:20:05.505 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:20:05.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:20:05.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:20:05.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:20:05.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:20:05.507 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:20:05.508 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:20:05.508 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:20:05.509 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:20:05.510 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:05.510 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:05.510 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:20:05.510 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:20:05.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T07:20:05.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:20:05.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:20:05.606 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:20:05.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:20:05.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:20:05.607 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:20:05.608 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:20:05.608 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:20:05.609 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:20:05.609 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:05.609 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:05.609 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:20:05.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:20:05.610 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T07:20:05.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2021-07-26T07:20:06.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T07:20:06.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:20:06.829 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:20:06.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:20:06.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T07:20:06.830 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:20:06.836 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:20:06.837 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:06.837 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:06.837 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:06.837 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:06.838 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:06.838 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:06.838 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:20:06.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:20:06.839 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: unfound_erasure_coded: seq 0 3 2021-07-26T07:20:06.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: unfound_erasure_coded: for id in $(seq 0 3) 2021-07-26T07:20:06.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: '[' false = true ']' 2021-07-26T07:20:06.839 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: run_osd_filestore td/osd-scrub-repair 0 2021-07-26T07:20:06.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T07:20:06.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T07:20:06.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2021-07-26T07:20:06.840 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T07:20:06.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:20:06.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:20:06.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:20:06.841 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:20:06.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:20:06.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:20:06.842 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:20:06.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:20:06.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T07:20:06.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T07:20:06.843 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:20:06.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T07:20:06.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:06.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:06.844 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:06.844 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:06.845 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:06.845 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:06.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:06.845 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T07:20:06.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T07:20:06.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T07:20:06.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:20:06.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:20:06.846 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:20:06.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:20:06.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:06.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T07:20:06.847 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T07:20:06.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:20:06.848 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T07:20:06.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=15fc017c-4733-466a-8ef0-7600f072308b 2021-07-26T07:20:06.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd1 15fc017c-4733-466a-8ef0-7600f072308b' 2021-07-26T07:20:06.849 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 15fc017c-4733-466a-8ef0-7600f072308b 2021-07-26T07:20:06.850 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T07:20:06.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQAmYv5gPWFUIhAAEpvA760vyzKmiZM84XxsEQ== 2021-07-26T07:20:06.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQAmYv5gPWFUIhAAEpvA760vyzKmiZM84XxsEQ=="}' 2021-07-26T07:20:06.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 15fc017c-4733-466a-8ef0-7600f072308b -i td/osd-scrub-repair/0/new.json 2021-07-26T07:20:06.879 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:20:06.890 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2021-07-26T07:20:06.891 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAmYv5gPWFUIhAAEpvA760vyzKmiZM84XxsEQ== --osd-uuid 15fc017c-4733-466a-8ef0-7600f072308b --osd-objectstore=filestore 2021-07-26T07:20:06.916 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:06.915+0000 7f0c0ae970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:06.918 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:06.917+0000 7f0c0ae970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:06.919 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:06.918+0000 7f0c0ae970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:07.390 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:07.390+0000 7f0c0ae970c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:07.920 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:07.918+0000 7f0c0ae970c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:07.923 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:07.918+0000 7f0c0ae970c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:20:07.923 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:07.918+0000 7f0c0ae970c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:20:08.400 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T07:20:08.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T07:20:08.401 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd0 key to auth repository 2021-07-26T07:20:08.402 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:20:08.402 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:20:08.576 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2021-07-26T07:20:08.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.0 2021-07-26T07:20:08.578 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:20:08.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:08.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T07:20:08.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T07:20:08.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T07:20:08.600 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:08.599+0000 7ffb2266f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:08.602 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:08.601+0000 7ffb2266f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:08.603 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:08.602+0000 7ffb2266f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:08.604 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:08.603+0000 7ffb2266f0c0 -1 Falling back to public interface 2021-07-26T07:20:08.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 0 2021-07-26T07:20:08.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:20:08.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:20:08.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:20:08.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:20:08.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:08.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:20:08.915 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:20:08.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:08.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:20:09.084 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:09.082+0000 7ffb2266f0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:09.084 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:09.083+0000 7ffb2266f0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:20:09.085 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:09.083+0000 7ffb2266f0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:20:09.092 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:09.091+0000 7ffb2266f0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:20:09.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:10.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:10.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:10.151 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:20:10.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:20:10.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:10.153 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:20:10.367 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/449494,v1:127.0.0.1:6801/449494] [v2:127.0.0.1:6802/449494,v1:127.0.0.1:6803/449494] exists,up 15fc017c-4733-466a-8ef0-7600f072308b 2021-07-26T07:20:10.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:20:10.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:20:10.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:20:10.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: unfound_erasure_coded: for id in $(seq 0 3) 2021-07-26T07:20:10.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: '[' false = true ']' 2021-07-26T07:20:10.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: run_osd_filestore td/osd-scrub-repair 1 2021-07-26T07:20:10.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T07:20:10.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T07:20:10.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2021-07-26T07:20:10.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T07:20:10.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:20:10.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:20:10.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:20:10.380 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:20:10.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:20:10.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:20:10.381 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:20:10.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:20:10.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T07:20:10.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T07:20:10.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:20:10.383 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T07:20:10.383 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:10.383 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:10.383 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:10.384 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:10.384 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:10.384 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:10.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:10.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T07:20:10.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T07:20:10.385 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T07:20:10.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:20:10.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:20:10.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:20:10.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:20:10.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:10.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T07:20:10.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T07:20:10.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:20:10.387 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T07:20:10.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=44b4c434-4eb4-4baa-905a-77c39b25e479 2021-07-26T07:20:10.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd1 44b4c434-4eb4-4baa-905a-77c39b25e479' 2021-07-26T07:20:10.388 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 44b4c434-4eb4-4baa-905a-77c39b25e479 2021-07-26T07:20:10.389 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T07:20:10.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQAqYv5g8bUCFxAAHIwc5OiF0R72eCtcxEp7xg== 2021-07-26T07:20:10.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQAqYv5g8bUCFxAAHIwc5OiF0R72eCtcxEp7xg=="}' 2021-07-26T07:20:10.390 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 44b4c434-4eb4-4baa-905a-77c39b25e479 -i td/osd-scrub-repair/1/new.json 2021-07-26T07:20:10.690 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:20:10.702 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2021-07-26T07:20:10.703 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAqYv5g8bUCFxAAHIwc5OiF0R72eCtcxEp7xg== --osd-uuid 44b4c434-4eb4-4baa-905a-77c39b25e479 --osd-objectstore=filestore 2021-07-26T07:20:10.728 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:10.727+0000 7fc27a4ce0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:10.740 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:10.740+0000 7fc27a4ce0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:10.741 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:10.740+0000 7fc27a4ce0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:11.324 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:11.323+0000 7fc27a4ce0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:12.199 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:12.199+0000 7fc27a4ce0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:12.207 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:12.207+0000 7fc27a4ce0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:20:12.207 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:12.207+0000 7fc27a4ce0c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:20:12.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T07:20:12.911 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T07:20:12.912 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd1 key to auth repository 2021-07-26T07:20:12.912 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T07:20:12.913 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:20:13.252 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T07:20:13.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.1 2021-07-26T07:20:13.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T07:20:13.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:13.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T07:20:13.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T07:20:13.274 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:13.273+0000 7f2b75c410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:13.296 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:13.295+0000 7f2b75c410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:13.297 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:13.296+0000 7f2b75c410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:13.298 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:13.298+0000 7f2b75c410c0 -1 Falling back to public interface 2021-07-26T07:20:13.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 1 2021-07-26T07:20:13.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:20:13.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:20:13.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:20:13.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:20:13.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:13.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:20:13.525 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:20:13.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:13.527 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:20:13.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:13.816 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:13.815+0000 7f2b75c410c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:13.816 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:13.815+0000 7f2b75c410c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:20:13.816 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:13.815+0000 7f2b75c410c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:20:13.821 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:13.821+0000 7f2b75c410c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:20:14.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:14.817 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:14.817 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:20:14.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:20:14.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:14.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:20:15.131 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/449867,v1:127.0.0.1:6811/449867] [v2:127.0.0.1:6812/449867,v1:127.0.0.1:6813/449867] exists,up 44b4c434-4eb4-4baa-905a-77c39b25e479 2021-07-26T07:20:15.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:20:15.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:20:15.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:20:15.137 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: unfound_erasure_coded: for id in $(seq 0 3) 2021-07-26T07:20:15.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: '[' false = true ']' 2021-07-26T07:20:15.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: run_osd_filestore td/osd-scrub-repair 2 2021-07-26T07:20:15.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T07:20:15.138 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T07:20:15.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2021-07-26T07:20:15.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T07:20:15.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:20:15.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:20:15.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:20:15.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:20:15.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:20:15.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T07:20:15.140 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T07:20:15.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:20:15.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T07:20:15.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T07:20:15.141 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:20:15.141 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T07:20:15.142 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:15.142 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:15.142 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:15.142 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:15.143 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:15.143 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:15.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:15.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T07:20:15.143 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T07:20:15.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T07:20:15.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:20:15.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:20:15.144 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:20:15.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:20:15.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:15.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T07:20:15.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T07:20:15.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2021-07-26T07:20:15.146 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T07:20:15.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=91f0aefe-1f5a-4430-9a9a-4fcbc61fee26 2021-07-26T07:20:15.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd1 91f0aefe-1f5a-4430-9a9a-4fcbc61fee26' 2021-07-26T07:20:15.147 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 91f0aefe-1f5a-4430-9a9a-4fcbc61fee26 2021-07-26T07:20:15.147 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T07:20:15.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQAvYv5g5MrqCBAAkQ7BZNM5DEQPKElZbKVPcQ== 2021-07-26T07:20:15.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQAvYv5g5MrqCBAAkQ7BZNM5DEQPKElZbKVPcQ=="}' 2021-07-26T07:20:15.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 91f0aefe-1f5a-4430-9a9a-4fcbc61fee26 -i td/osd-scrub-repair/2/new.json 2021-07-26T07:20:15.517 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:20:15.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2021-07-26T07:20:15.529 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAvYv5g5MrqCBAAkQ7BZNM5DEQPKElZbKVPcQ== --osd-uuid 91f0aefe-1f5a-4430-9a9a-4fcbc61fee26 --osd-objectstore=filestore 2021-07-26T07:20:15.553 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:15.553+0000 7f76fea380c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:15.558 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:15.558+0000 7f76fea380c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:15.559 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:15.558+0000 7f76fea380c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:16.113 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:16.113+0000 7f76fea380c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:16.601 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:16.600+0000 7f76fea380c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:16.616 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:16.616+0000 7f76fea380c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:20:16.617 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:16.616+0000 7f76fea380c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:20:17.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T07:20:17.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T07:20:17.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd2 key to auth repository 2021-07-26T07:20:17.187 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T07:20:17.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:20:17.518 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2021-07-26T07:20:17.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.2 2021-07-26T07:20:17.520 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T07:20:17.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T07:20:17.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:17.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T07:20:17.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T07:20:17.543 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:17.542+0000 7fb5b7e860c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:17.544 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:17.544+0000 7fb5b7e860c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:17.545 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:17.545+0000 7fb5b7e860c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:17.546 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:17.546+0000 7fb5b7e860c0 -1 Falling back to public interface 2021-07-26T07:20:17.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 2 2021-07-26T07:20:17.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:20:17.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:20:17.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:20:17.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:20:17.854 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:17.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:20:17.855 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:20:17.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:17.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:20:18.000 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:17.999+0000 7fb5b7e860c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:18.000 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:17.999+0000 7fb5b7e860c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:20:18.000 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:17.999+0000 7fb5b7e860c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:20:18.005 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:18.004+0000 7fb5b7e860c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T07:20:18.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:19.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:19.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:19.165 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:20:19.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:20:19.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:19.166 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:20:19.466 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/450150,v1:127.0.0.1:6819/450150] [v2:127.0.0.1:6820/450150,v1:127.0.0.1:6821/450150] exists,up 91f0aefe-1f5a-4430-9a9a-4fcbc61fee26 2021-07-26T07:20:19.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:20:19.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:20:19.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:20:19.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: unfound_erasure_coded: for id in $(seq 0 3) 2021-07-26T07:20:19.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: '[' false = true ']' 2021-07-26T07:20:19.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: run_osd_filestore td/osd-scrub-repair 3 2021-07-26T07:20:19.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2021-07-26T07:20:19.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T07:20:19.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=3 2021-07-26T07:20:19.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T07:20:19.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/3 2021-07-26T07:20:19.474 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:20:19.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:20:19.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:20:19.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:20:19.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T07:20:19.475 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T07:20:19.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:20:19.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T07:20:19.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T07:20:19.476 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:20:19.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T07:20:19.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:19.477 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:19.477 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:19.477 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:19.478 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:19.478 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:19.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:19.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T07:20:19.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T07:20:19.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T07:20:19.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:20:19.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:20:19.479 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:20:19.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:20:19.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:19.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T07:20:19.480 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T07:20:19.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-scrub-repair/3 2021-07-26T07:20:19.481 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T07:20:19.481 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=c05811d1-713e-40ae-93c5-3d665bfd9f2d 2021-07-26T07:20:19.482 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 c05811d1-713e-40ae-93c5-3d665bfd9f2d 2021-07-26T07:20:19.482 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd1 c05811d1-713e-40ae-93c5-3d665bfd9f2d' 2021-07-26T07:20:19.483 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T07:20:19.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQAzYv5gwF7ZHBAAxh6VhNJCREoErLzXuMdvFQ== 2021-07-26T07:20:19.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQAzYv5gwF7ZHBAAxh6VhNJCREoErLzXuMdvFQ=="}' 2021-07-26T07:20:19.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new c05811d1-713e-40ae-93c5-3d665bfd9f2d -i td/osd-scrub-repair/3/new.json 2021-07-26T07:20:19.843 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:20:19.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-scrub-repair/3/new.json 2021-07-26T07:20:19.856 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAzYv5gwF7ZHBAAxh6VhNJCREoErLzXuMdvFQ== --osd-uuid c05811d1-713e-40ae-93c5-3d665bfd9f2d --osd-objectstore=filestore 2021-07-26T07:20:19.880 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:19.879+0000 7f83532f10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:19.885 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:19.884+0000 7f83532f10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:19.885 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:19.885+0000 7f83532f10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:20.422 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:20.422+0000 7f83532f10c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:20.909 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:20.908+0000 7f83532f10c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:20.909 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:20.908+0000 7f83532f10c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:20:20.909 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:20.908+0000 7f83532f10c0 -1 journal do_read_entry(4096): bad header magic 2021-07-26T07:20:21.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-scrub-repair/3/keyring 2021-07-26T07:20:21.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T07:20:21.562 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd3 key to auth repository 2021-07-26T07:20:21.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd3 key to auth repository 2021-07-26T07:20:21.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:20:21.891 INFO:tasks.workunit.client.0.smithi071.stderr:Error EINVAL: key for osd.3 not found in provided keyring 2021-07-26T07:20:21.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.3 2021-07-26T07:20:21.893 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.3 2021-07-26T07:20:21.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T07:20:21.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:21.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T07:20:21.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T07:20:21.915 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:21.914+0000 7ff26289f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:21.916 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:21.916+0000 7ff26289f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:21.917 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:21.917+0000 7ff26289f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:21.919 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:21.918+0000 7ff26289f0c0 -1 Falling back to public interface 2021-07-26T07:20:22.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 3 2021-07-26T07:20:22.278 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:20:22.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T07:20:22.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:20:22.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:20:22.279 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:22.280 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:20:22.280 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:20:22.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:22.281 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:20:22.383 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:22.382+0000 7ff26289f0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:22.401 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:22.400+0000 7ff26289f0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:20:22.401 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:22.400+0000 7ff26289f0c0 -1 journal do_read_entry(8192): bad header magic 2021-07-26T07:20:22.406 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:22.405+0000 7ff26289f0c0 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T07:20:22.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:23.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:23.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:23.546 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:20:23.547 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:20:23.547 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:23.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:20:23.848 INFO:tasks.workunit.client.0.smithi071.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/450441,v1:127.0.0.1:6827/450441] [v2:127.0.0.1:6828/450441,v1:127.0.0.1:6829/450441] exists,up c05811d1-713e-40ae-93c5-3d665bfd9f2d 2021-07-26T07:20:23.850 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:20:23.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:20:23.851 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:20:23.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:872: unfound_erasure_coded: create_ec_pool ecpool false k=2 m=2 2021-07-26T07:20:23.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=ecpool 2021-07-26T07:20:23.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T07:20:23.852 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=false 2021-07-26T07:20:23.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T07:20:23.853 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2021-07-26T07:20:24.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2021-07-26T07:20:24.367 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2021-07-26T07:20:25.301 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' created 2021-07-26T07:20:25.311 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:20:26.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' false = true ']' 2021-07-26T07:20:26.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T07:20:26.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:20:26.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:20:26.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:20:26.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:20:26.314 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:20:26.316 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:20:26.316 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:20:26.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:20:26.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:20:26.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:20:26.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:20:26.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:20:26.423 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:20:26.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:20:26.424 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:20:26.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:20:26.692 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:20:26.692 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:20:26.692 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T07:20:26.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:20:26.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:26.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:20:26.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836484 2021-07-26T07:20:26.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836484 2021-07-26T07:20:26.810 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484' 2021-07-26T07:20:26.811 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:26.811 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:20:26.927 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672964 2021-07-26T07:20:26.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672964 2021-07-26T07:20:26.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484 1-42949672964' 2021-07-26T07:20:26.928 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:26.928 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:20:27.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509443 2021-07-26T07:20:27.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509443 2021-07-26T07:20:27.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484 1-42949672964 2-64424509443' 2021-07-26T07:20:27.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:27.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:20:27.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345923 2021-07-26T07:20:27.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345923 2021-07-26T07:20:27.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836484 1-42949672964 2-64424509443 3-85899345923' 2021-07-26T07:20:27.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:27.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836484 2021-07-26T07:20:27.165 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:27.165 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:20:27.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836484 2021-07-26T07:20:27.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:27.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836484 2021-07-26T07:20:27.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2021-07-26T07:20:27.168 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836484 2021-07-26T07:20:27.168 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:20:27.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836485 -lt 21474836484 2021-07-26T07:20:27.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:27.458 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672964 2021-07-26T07:20:27.459 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:27.459 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:20:27.460 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672964 2021-07-26T07:20:27.460 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:27.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672964 2021-07-26T07:20:27.461 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2021-07-26T07:20:27.461 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672964 2021-07-26T07:20:27.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:20:27.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672964 -lt 42949672964 2021-07-26T07:20:27.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:27.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509443 2021-07-26T07:20:27.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:27.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:20:27.782 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509443 2021-07-26T07:20:27.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:27.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509443 2021-07-26T07:20:27.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2021-07-26T07:20:27.784 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509443 2021-07-26T07:20:27.784 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:20:28.050 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509443 -lt 64424509443 2021-07-26T07:20:28.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:28.051 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345923 2021-07-26T07:20:28.052 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:28.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:20:28.053 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345923 2021-07-26T07:20:28.053 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:28.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345923 2021-07-26T07:20:28.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345923' 2021-07-26T07:20:28.054 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.3 seq 85899345923 2021-07-26T07:20:28.055 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:20:28.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345923 -lt 85899345923 2021-07-26T07:20:28.332 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:20:28.333 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:28.333 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:28.725 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:20:28.726 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:20:28.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:20:28.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:20:28.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:20:28.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:20:28.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:20:28.727 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:20:29.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:20:29.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:20:29.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:29.063 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:29.386 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:20:29.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:20:29.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:20:29.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T07:20:29.387 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:874: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2021-07-26T07:20:29.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:20:29.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T07:20:29.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2021-07-26T07:20:29.388 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:20:29.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:20:29.389 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:20:30.138 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:20:30.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:20:30.683 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:20:30.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:20:30.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:20:30.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T07:20:30.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: get_primary ecpool SOMETHING 2021-07-26T07:20:30.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=ecpool 2021-07-26T07:20:30.844 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T07:20:30.845 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T07:20:30.845 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:20:31.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: local primary=1 2021-07-26T07:20:31.118 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2021-07-26T07:20:31.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: get_osds ecpool SOMETHING 2021-07-26T07:20:31.119 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: sed -e s/1// 2021-07-26T07:20:31.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=ecpool 2021-07-26T07:20:31.120 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=SOMETHING 2021-07-26T07:20:31.120 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map ecpool SOMETHING 2021-07-26T07:20:31.120 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:20:31.428 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=1 2021-07-26T07:20:31.428 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:20:31.428 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:20:31.429 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T07:20:31.429 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 1 0 2 3 2021-07-26T07:20:31.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: local -a osds 2021-07-26T07:20:31.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: local not_primary_first=0 2021-07-26T07:20:31.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: local not_primary_second=2 2021-07-26T07:20:31.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: local not_primary_third=3 2021-07-26T07:20:31.433 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:885: unfound_erasure_coded: pids= 2021-07-26T07:20:31.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:886: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T07:20:31.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T07:20:31.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T07:20:31.435 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 451171"' 2021-07-26T07:20:31.436 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 451171' 2021-07-26T07:20:31.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T07:20:31.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T07:20:31.436 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T07:20:31.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 451172"' 2021-07-26T07:20:31.437 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 451172' 2021-07-26T07:20:31.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T07:20:31.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T07:20:31.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T07:20:31.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T07:20:31.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/451174: /' 2021-07-26T07:20:31.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 451175"' 2021-07-26T07:20:31.439 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 451175' 2021-07-26T07:20:31.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T07:20:31.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:889: unfound_erasure_coded: wait_background pids 2021-07-26T07:20:31.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 451171 451172 451175' 2021-07-26T07:20:31.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T07:20:31.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T07:20:31.440 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 451171 2021-07-26T07:20:31.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/451177: /' 2021-07-26T07:20:31.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T07:20:31.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/451180: /' 2021-07-26T07:20:33.097 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:20:33.097 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:20:33.098 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=2 2021-07-26T07:20:33.098 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:20:33.098 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T07:20:33.098 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:20:33.099 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:20:33.099 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=2 2021-07-26T07:20:33.099 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:20:33.099 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2021-07-26T07:20:33.100 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:20:33.100 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:20:33.100 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:20:33.100 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:20:33.100 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:20:33.101 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:20:33.101 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T07:20:33.101 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:20:33.101 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:20:33.102 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=2 2021-07-26T07:20:33.102 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:20:33.102 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:20:33.102 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2021-07-26T07:20:33.103 INFO:tasks.workunit.client.0.smithi071.stderr:451177: Error getting attr on : 1.0s2_head,2#1:00000000::::head#, (61) No data available 2021-07-26T07:20:33.103 INFO:tasks.workunit.client.0.smithi071.stderr:451177: remove 2#1:eb822e21:::SOMETHING:head# 2021-07-26T07:20:33.103 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2021-07-26T07:20:33.103 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:20:33.103 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:20:33.104 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T07:20:33.104 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:20:33.104 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:20:33.104 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:20:33.105 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ra451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:20:33.105 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:20:33.105 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T07:20:33.105 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:20:33.105 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T07:20:33.106 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:20:33.106 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:20:33.106 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T07:20:33.107 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:20:33.107 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T07:20:33.107 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:20:33.107 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:20:33.108 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:20:33.108 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:20:33.108 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:20:33.108 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:20:33.108 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T07:20:33.109 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:20:33.109 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:20:33.109 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T07:20:33.109 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:20:33.109 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T07:20:33.110 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2021-07-26T07:20:33.110 INFO:tasks.workunit.client.0.smithi071.stderr:451180: Error getting attr on : 1.0s3_head,3#1:00000000::::head#, (61) No data available 2021-07-26T07:20:33.110 INFO:tasks.workunit.client.0.smithi071.stderr:451180: remove 3#1:eb822e21:::SOMETHING:head# 2021-07-26T07:20:33.111 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T07:20:33.111 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:20:33.111 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:20:33.111 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T07:20:33.112 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:20:33.114 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T07:20:33.114 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:20:33.114 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ra451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:20:33.115 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:20:33.115 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:20:33.115 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:20:33.115 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T07:20:33.115 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:20:33.116 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:20:33.116 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:20:33.116 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:20:33.117 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:20:33.117 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:20:33.117 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:20:33.117 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:20:33.118 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:20:33.118 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:20:33.118 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:20:33.118 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T07:20:33.118 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:20:33.119 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:20:33.119 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:20:33.119 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:20:33.119 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:20:33.119 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2021-07-26T07:20:33.120 INFO:tasks.workunit.client.0.smithi071.stderr:451174: Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:20:33.120 INFO:tasks.workunit.client.0.smithi071.stderr:451174: remove 1#1:eb822e21:::SOMETHING:head# 2021-07-26T07:20:33.120 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:20:33.120 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:20:33.121 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:20:33.121 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:20:33.121 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:20:33.121 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:20:33.122 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:20:33.123 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:20:33.124 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:20:33.124 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:20:33.125 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T07:20:33.125 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T07:20:33.125 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:20:33.125 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:20:33.126 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:20:33.126 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:20:33.127 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:20:33.127 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:33.127 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:33.127 INFO:tasks.workunit.client.0.smithi071.stderr:451177: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:33.128 INFO:tasks.workunit.client.0.smithi071.stderr:451177: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:33.128 INFO:tasks.workunit.client.0.smithi071.stderr:451177: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:33.128 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:33.128 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:33.129 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:20:33.129 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:20:33.129 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:20:33.130 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:20:33.130 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:20:33.130 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:33.130 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:20:33.131 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:20:33.131 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T07:20:33.131 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T07:20:33.131 INFO:tasks.workunit.client.0.smithi071.stderr:451177: start osd.2 2021-07-26T07:20:33.132 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-extio=.99' 2021-07-26T07:20:33.132 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:20:33.132 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:20:33.132 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:20:33.133 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:20:33.133 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:20:33.133 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:20:33.133 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:20:33.133 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:20:33.134 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:20:33.134 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:33.134 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:33.135 INFO:tasks.workunit.client.0.smithi071.stderr:451174: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:33.135 INFO:tasks.workunit.client.0.smithi071.stderr:451174: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:33.136 INFO:tasks.workunit.client.0.smithi071.stderr:451174: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:33.136 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:33.136 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:33.136 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:20:33.137 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:20:33.137 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:20:33.137 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:20:33.137 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:20:33.138 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:33.138 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:20:33.138 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:20:33.138 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:20:33.138 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:20:33.139 INFO:tasks.workunit.client.0.smithi071.stderr:451174: start osd.0 2021-07-26T07:20:33.139 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-extio=.99' 2021-07-26T07:20:33.139 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:20:33.139 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:20:33.140 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T07:20:33.140 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T07:20:33.140 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:20:33.140 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:20:33.141 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:20:33.141 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:20:33.141 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:20:33.141 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:33.142 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:33.142 INFO:tasks.workunit.client.0.smithi071.stderr:451180: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:33.142 INFO:tasks.workunit.client.0.smithi071.stderr:451180: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:33.142 INFO:tasks.workunit.client.0.smithi071.stderr:451180: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:33.143 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:33.143 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:33.143 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:20:33.143 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:20:33.144 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:20:33.144 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:20:33.144 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:20:33.144 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:33.145 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:20:33.145 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:20:33.145 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T07:20:33.145 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T07:20:33.146 INFO:tasks.workunit.client.0.smithi071.stderr:451180: start osd.3 2021-07-26T07:20:35.014 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:35.015 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:20:35.015 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:20:35.016 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:20:35.016 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:20:35.016 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:20:35.017 INFO:tasks.workunit.client.0.smithi071.stderr:451174: 2021-07-26T07:20:33.125+0000 7f99425960c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:35.017 INFO:tasks.workunit.client.0.smithi071.stderr:451174: 2021-07-26T07:20:33.128+0000 7f99425960c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:35.017 INFO:tasks.workunit.client.0.smithi071.stderr:451174: 2021-07-26T07:20:33.129+0000 7f99425960c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:35.017 INFO:tasks.workunit.client.0.smithi071.stderr:451174: 2021-07-26T07:20:33.131+0000 7f99425960c0 -1 Falling back to public interface 2021-07-26T07:20:35.018 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:20:35.018 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:20:35.018 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:20:35.018 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:20:35.019 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:20:35.019 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:35.019 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:20:35.019 INFO:tasks.workunit.client.0.smithi071.stderr:451174: 0 2021-07-26T07:20:35.019 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:35.020 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:20:35.020 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:35.020 INFO:tasks.workunit.client.0.smithi071.stderr:451174: 2021-07-26T07:20:33.798+0000 7f99425960c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:35.020 INFO:tasks.workunit.client.0.smithi071.stderr:451174: 2021-07-26T07:20:33.807+0000 7f99425960c0 -1 journal do_read_entry(196608): bad header magic 2021-07-26T07:20:35.021 INFO:tasks.workunit.client.0.smithi071.stderr:451174: 2021-07-26T07:20:33.807+0000 7f99425960c0 -1 journal do_read_entry(196608): bad header magic 2021-07-26T07:20:35.021 INFO:tasks.workunit.client.0.smithi071.stderr:451174: 2021-07-26T07:20:33.814+0000 7f99425960c0 -1 osd.0 28 log_to_monitors {default=true} 2021-07-26T07:20:35.021 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:35.021 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:35.022 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:20:35.022 INFO:tasks.workunit.client.0.smithi071.stderr:451174: 1 2021-07-26T07:20:35.022 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:35.022 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:20:35.022 INFO:tasks.workunit.client.0.smithi071.stderr:451174: osd.0 up in weight 1 up_from 33 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6801/451382,v1:127.0.0.1:6803/451382] [v2:127.0.0.1:6805/451382,v1:127.0.0.1:6807/451382] exists,up 15fc017c-4733-466a-8ef0-7600f072308b 2021-07-26T07:20:35.023 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:20:35.023 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:20:35.024 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:20:35.024 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:20:35.024 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:20:35.024 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:20:35.025 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.perimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:35.025 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/2/whoami 2021-07-26T07:20:35.025 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T07:20:35.026 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:20:35.026 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:20:35.026 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:20:35.026 INFO:tasks.workunit.client.0.smithi071.stderr:451177: 2021-07-26T07:20:33.137+0000 7f7edda1f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:35.027 INFO:tasks.workunit.client.0.smithi071.stderr:451177: 2021-07-26T07:20:33.140+0000 7f7edda1f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:35.027 INFO:tasks.workunit.client.0.smithi071.stderr:451177: 2021-07-26T07:20:33.141+0000 7f7edda1f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:35.027 INFO:tasks.workunit.client.0.smithi071.stderr:451177: 2021-07-26T07:20:33.142+0000 7f7edda1f0c0 -1 Falling back to public interface 2021-07-26T07:20:35.027 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T07:20:35.027 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:20:35.028 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:20:35.028 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:20:35.028 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:20:35.028 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:35.029 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:20:35.029 INFO:tasks.workunit.client.0.smithi071.stderr:451177: 0 2021-07-26T07:20:35.029 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:35.029 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:20:35.029 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:35.030 INFO:tasks.workunit.client.0.smithi071.stderr:451177: 2021-07-26T07:20:33.815+0000 7f7edda1f0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:35.030 INFO:tasks.workunit.client.0.smithi071.stderr:451177: 2021-07-26T07:20:33.828+0000 7f7edda1f0c0 -1 journal do_read_entry(192512): bad header magic 2021-07-26T07:20:35.030 INFO:tasks.workunit.client.0.smithi071.stderr:451177: 2021-07-26T07:20:33.828+0000 7f7edda1f0c0 -1 journal do_read_entry(192512): bad header magic 2021-07-26T07:20:35.031 INFO:tasks.workunit.client.0.smithi071.stderr:451177: 2021-07-26T07:20:33.834+0000 7f7edda1f0c0 -1 osd.2 28 log_to_monitors {default=true} 2021-07-26T07:20:35.031 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:35.031 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:35.031 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:20:35.032 INFO:tasks.workunit.client.0.smithi071.stderr:451177: 1 2021-07-26T07:20:35.032 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:35.032 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:20:35.032 INFO:tasks.workunit.client.0.smithi071.stderr:451177: osd.2 up in weight 1 up_from 33 up_thru 0 down_at 30 last_clean_interval [15,28) [v2:127.0.0.1:6826/451426,v1:127.0.0.1:6827/451426] [v2:127.0.0.1:6828/451426,v1:127.0.0.1:6829/451426] exists,up 91f0aefe-1f5a-4430-9a9a-4fcbc61fee26 2021-07-26T07:20:35.033 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:20:35.033 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:20:35.033 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:20:35.033 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:20:35.033 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:20:35.034 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:20:35.034 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/cloneperimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:35.035 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T07:20:35.035 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T07:20:35.035 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:20:35.035 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:20:35.036 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:20:35.036 INFO:tasks.workunit.client.0.smithi071.stderr:451180: 2021-07-26T07:20:33.125+0000 7fb4b4a760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:35.036 INFO:tasks.workunit.client.0.smithi071.stderr:451180: 2021-07-26T07:20:33.128+0000 7fb4b4a760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:35.036 INFO:tasks.workunit.client.0.smithi071.stderr:451180: 2021-07-26T07:20:33.129+0000 7fb4b4a760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:35.037 INFO:tasks.workunit.client.0.smithi071.stderr:451180: 2021-07-26T07:20:33.131+0000 7fb4b4a760c0 -1 Falling back to public interface 2021-07-26T07:20:35.037 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T07:20:35.037 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:20:35.037 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T07:20:35.038 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:20:35.038 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:20:35.038 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:35.038 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:20:35.039 INFO:tasks.workunit.client.0.smithi071.stderr:451180: 0 2021-07-26T07:20:35.039 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:35.039 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:20:35.039 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:35.040 INFO:tasks.workunit.client.0.smithi071.stderr:451180: 2021-07-26T07:20:33.815+0000 7fb4b4a760c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T07:20:35.040 INFO:tasks.workunit.client.0.smithi071.stderr:451180: 2021-07-26T07:20:33.828+0000 7fb4b4a760c0 -1 journal do_read_entry(163840): bad header magic 2021-07-26T07:20:35.040 INFO:tasks.workunit.client.0.smithi071.stderr:451180: 2021-07-26T07:20:33.828+0000 7fb4b4a760c0 -1 journal do_read_entry(163840): bad header magic 2021-07-26T07:20:35.041 INFO:tasks.workunit.client.0.smithi071.stderr:451180: 2021-07-26T07:20:33.834+0000 7fb4b4a760c0 -1 osd.3 28 log_to_monitors {default=true} 2021-07-26T07:20:35.041 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:35.041 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:35.041 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:20:35.042 INFO:tasks.workunit.client.0.smithi071.stderr:451180: 1 2021-07-26T07:20:35.042 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:35.042 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:20:35.043 INFO:tasks.workunit.client.0.smithi071.stderr:451180: osd.3 up in weight 1 up_from 33 up_thru 0 down_at 29 last_clean_interval [20,28) [v2:127.0.0.1:6800/451383,v1:127.0.0.1:6802/451383] [v2:127.0.0.1:6804/451383,v1:127.0.0.1:6806/451383] exists,up c05811d1-713e-40ae-93c5-3d665bfd9f2d 2021-07-26T07:20:35.043 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:20:35.043 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:20:35.043 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:20:35.044 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:20:35.044 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:20:35.044 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:20:35.998 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:20:35.999 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:20:36.000 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:20:36.000 INFO:tasks.workunit.client.0.smithi071.stderr:451177: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:20:36.000 INFO:tasks.workunit.client.0.smithi071.stderr:451177: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:20:36.000 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:20:36.000 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:20:36.001 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:20:36.001 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:20:36.001 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:20:36.001 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:20:36.002 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:20:36.002 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:20:36.002 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:20:36.002 INFO:tasks.workunit.client.0.smithi071.stderr:451177: 1 2021-07-26T07:20:36.002 INFO:tasks.workunit.client.0.smithi071.stderr:451177: 2 2021-07-26T07:20:36.003 INFO:tasks.workunit.client.0.smithi071.stderr:451177: 3' 2021-07-26T07:20:36.003 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:20:36.003 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:36.003 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:20:36.004 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=141733920771 2021-07-26T07:20:36.004 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 141733920771 2021-07-26T07:20:36.004 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-141733920771' 2021-07-26T07:20:36.004 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:36.004 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:20:36.005 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672969 2021-07-26T07:20:36.005 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672969 2021-07-26T07:20:36.005 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-141733920771 1-42949672969' 2021-07-26T07:20:36.007 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:36.007 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:20:36.008 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=141733920772 2021-07-26T07:20:36.008 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 141733920772 2021-07-26T07:20:36.008 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-141733920771 1-42949672969 2-141733920772' 2021-07-26T07:20:36.009 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:36.009 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:20:36.009 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=141733920770 2021-07-26T07:20:36.010 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 141733920770 2021-07-26T07:20:36.011 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntclient.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:20:36.011 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:20:36.011 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:20:36.012 INFO:tasks.workunit.client.0.smithi071.stderr:451174: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:20:36.012 INFO:tasks.workunit.client.0.smithi071.stderr:451174: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:20:36.012 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:20:36.012 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:20:36.012 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:20:36.013 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:20:36.013 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:20:36.013 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:20:36.013 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:20:36.014 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:20:36.014 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:20:36.014 INFO:tasks.workunit.client.0.smithi071.stderr:451174: 1 2021-07-26T07:20:36.015 INFO:tasks.workunit.client.0.smithi071.stderr:451174: 2 2021-07-26T07:20:36.015 INFO:tasks.workunit.client.0.smithi071.stderr:451174: 3' 2021-07-26T07:20:36.015 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:20:36.016 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:36.016 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:20:36.017 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=141733920772 2021-07-26T07:20:36.018 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 141733920772 2021-07-26T07:20:36.018 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-141733920772' 2021-07-26T07:20:36.018 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:36.020 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:20:36.020 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672968 2021-07-26T07:20:36.020 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672968 2021-07-26T07:20:36.021 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-141733920772 1-42949672968' 2021-07-26T07:20:36.021 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:36.022 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:20:36.022 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=141733920771 2021-07-26T07:20:36.022 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 141733920771 2021-07-26T07:20:36.022 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-141733920772 1-42949672968 2-141733920771' 2021-07-26T07:20:36.023 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:36.023 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:20:36.023 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=141733920771 2021-07-26T07:20:36.023 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 141733920771 2021-07-26T07:20:36.023 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:20:36.024 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:20:36.025 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:20:36.025 INFO:tasks.workunit.client.0.smithi071.stderr:451180: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:20:36.025 INFO:tasks.workunit.client.0.smithi071.stderr:451180: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:20:36.025 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:20:36.026 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:20:36.026 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:20:36.026 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:20:36.026 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:20:36.027 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:20:36.027 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:20:36.027 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:20:36.027 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:20:36.027 INFO:tasks.workunit.client.0.smithi071.stderr:451180: 1 2021-07-26T07:20:36.028 INFO:tasks.workunit.client.0.smithi071.stderr:451180: 2 2021-07-26T07:20:36.028 INFO:tasks.workunit.client.0.smithi071.stderr:451180: 3' 2021-07-26T07:20:36.028 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:20:36.028 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:36.029 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:20:36.029 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=141733920770 2021-07-26T07:20:36.029 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 141733920770 2021-07-26T07:20:36.029 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-141733920770' 2021-07-26T07:20:36.030 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:36.030 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:20:36.030 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672967 2021-07-26T07:20:36.030 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672967 2021-07-26T07:20:36.030 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-141733920770 1-42949672967' 2021-07-26T07:20:36.031 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:36.031 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:20:36.031 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=141733920770 2021-07-26T07:20:36.033 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 141733920770 2021-07-26T07:20:36.033 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-141733920770 1-42949672967 2-141733920770' 2021-07-26T07:20:36.033 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:20:36.034 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:20:36.034 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=141733920772 2021-07-26T07:20:36.034 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 141733920772 2021-07-26T07:20:37.911 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-141733920771 1-42949672969 2-141733920772 3-141733920770' 2021-07-26T07:20:37.912 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:37.912 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-141733920771 2021-07-26T07:20:37.912 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:37.913 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:20:37.913 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-141733920771 2021-07-26T07:20:37.913 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:37.913 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=141733920771 2021-07-26T07:20:37.914 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 141733920771' 2021-07-26T07:20:37.914 INFO:tasks.workunit.client.0.smithi071.stderr:451177: waiting osd.0 seq 141733920771 2021-07-26T07:20:37.914 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:20:37.914 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 141733920771 2021-07-26T07:20:37.915 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:20:37.915 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:20:37.916 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:20:37.916 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 141733920772 -lt 141733920771 2021-07-26T07:20:37.916 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:37.917 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672969 2021-07-26T07:20:37.917 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:37.919 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:20:37.919 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672969 2021-07-26T07:20:37.920 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:37.920 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672969 2021-07-26T07:20:37.921 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-07-26T07:20:37.921 INFO:tasks.workunit.client.0.smithi071.stderr:451177: waiting osd.1 seq 42949672969 2021-07-26T07:20:37.921 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:20:37.922 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672969 -lt 42949672969 2021-07-26T07:20:37.922 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:37.923 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-141733920772 2021-07-26T07:20:37.923 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:37.923 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:20:37.923 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-141733920772 2021-07-26T07:20:37.923 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:37.924 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=141733920772 2021-07-26T07:20:37.924 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 141733920772' 2021-07-26T07:20:37.924 INFO:tasks.workunit.client.0.smithi071.stderr:451177: waiting osd.2 seq 141733920772 2021-07-26T07:20:37.925 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ce/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-141733920772 1-42949672968 2-141733920771 3-141733920771' 2021-07-26T07:20:37.925 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:37.925 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-141733920772 2021-07-26T07:20:37.925 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:37.926 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:20:37.926 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-141733920772 2021-07-26T07:20:37.926 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:37.926 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=141733920772 2021-07-26T07:20:37.927 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 141733920772' 2021-07-26T07:20:37.927 INFO:tasks.workunit.client.0.smithi071.stderr:451174: waiting osd.0 seq 141733920772 2021-07-26T07:20:37.927 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:20:37.927 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 141733920772 2021-07-26T07:20:37.928 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:20:37.928 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:20:37.928 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:20:37.928 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 141733920772 -lt 141733920772 2021-07-26T07:20:37.929 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:37.929 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672968 2021-07-26T07:20:37.929 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:37.929 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:20:37.929 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672968 2021-07-26T07:20:37.930 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:37.930 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672968 2021-07-26T07:20:37.930 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-07-26T07:20:37.930 INFO:tasks.workunit.client.0.smithi071.stderr:451174: waiting osd.1 seq 42949672968 2021-07-26T07:20:37.931 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:20:37.931 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672969 -lt 42949672968 2021-07-26T07:20:37.931 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:37.931 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-141733920771 2021-07-26T07:20:37.931 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:37.932 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:20:37.932 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-141733920771 2021-07-26T07:20:37.932 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:37.933 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=141733920771 2021-07-26T07:20:37.933 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2021-07-26T07:20:37.933 INFO:tasks.workunit.client.0.smithi071.stderr:451174: waiting osd.2 seq 141733920771 2021-07-26T07:20:37.972 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/cepu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-141733920770 1-42949672967 2-141733920770 3-141733920772' 2021-07-26T07:20:37.973 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:37.973 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-141733920770 2021-07-26T07:20:37.974 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:37.974 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:20:37.974 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-141733920770 2021-07-26T07:20:37.974 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:37.975 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=141733920770 2021-07-26T07:20:37.975 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 141733920770' 2021-07-26T07:20:37.976 INFO:tasks.workunit.client.0.smithi071.stderr:451180: waiting osd.0 seq 141733920770 2021-07-26T07:20:37.976 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:20:37.976 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 141733920770 2021-07-26T07:20:37.976 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:20:37.976 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:20:37.977 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:20:37.977 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 141733920772 -lt 141733920770 2021-07-26T07:20:37.977 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:37.978 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672967 2021-07-26T07:20:37.978 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:37.979 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:20:37.979 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672967 2021-07-26T07:20:37.979 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:37.979 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672967 2021-07-26T07:20:37.980 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2021-07-26T07:20:37.980 INFO:tasks.workunit.client.0.smithi071.stderr:451180: waiting osd.1 seq 42949672967 2021-07-26T07:20:37.980 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:20:37.980 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672969 -lt 42949672967 2021-07-26T07:20:37.981 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:37.981 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-141733920770 2021-07-26T07:20:37.981 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:37.981 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:20:37.981 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-141733920770 2021-07-26T07:20:37.982 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:37.982 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=141733920770 2021-07-26T07:20:37.982 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 141733920770' 2021-07-26T07:20:37.982 INFO:tasks.workunit.client.0.smithi071.stderr:451180: waiting osd.2 seq 141733920770 2021-07-26T07:20:39.587 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceh-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:20:39.587 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 141733920772 -lt 141733920771 2021-07-26T07:20:39.588 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:39.588 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-141733920771 2021-07-26T07:20:39.588 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:39.588 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:20:39.589 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-141733920771 2021-07-26T07:20:39.589 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:39.589 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=141733920771 2021-07-26T07:20:39.589 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 141733920771' 2021-07-26T07:20:39.589 INFO:tasks.workunit.client.0.smithi071.stderr:451174: waiting osd.3 seq 141733920771 2021-07-26T07:20:39.590 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:20:39.590 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 141733920772 -lt 141733920771 2021-07-26T07:20:39.590 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:20:39.590 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:39.591 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:39.591 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:20:39.591 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:20:39.591 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:20:39.592 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:20:39.592 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:20:39.593 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:20:39.593 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:20:39.594 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:20:39.594 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:20:39.595 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:20:39.595 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:39.595 INFO:tasks.workunit.client.0.smithi071.stderr:451174: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:39.595 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:20:39.596 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:20:39.596 INFO:tasks.workunit.client.0.smithi071.stderr:451174: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:20:39.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T07:20:39.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T07:20:39.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 451172 2021-07-26T07:20:39.597 INFO:tasks.workunit.client.0.smithi071.stderr:ph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:20:39.597 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 141733920772 -lt 141733920770 2021-07-26T07:20:39.598 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:39.598 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-141733920772 2021-07-26T07:20:39.598 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:39.599 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:20:39.599 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-141733920772 2021-07-26T07:20:39.599 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:39.599 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=141733920772 2021-07-26T07:20:39.600 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 141733920772' 2021-07-26T07:20:39.600 INFO:tasks.workunit.client.0.smithi071.stderr:451180: waiting osd.3 seq 141733920772 2021-07-26T07:20:39.600 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:20:39.600 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 141733920772 -lt 141733920772 2021-07-26T07:20:39.601 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:20:39.601 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:39.601 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:39.601 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:20:39.601 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:20:39.602 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:20:39.602 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:20:39.602 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:20:39.602 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:20:39.603 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:20:39.603 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:20:39.603 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:20:39.604 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:20:39.604 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:39.604 INFO:tasks.workunit.client.0.smithi071.stderr:451180: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:39.605 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:20:39.605 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:20:39.605 INFO:tasks.workunit.client.0.smithi071.stderr:451180: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:20:39.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T07:20:39.608 INFO:tasks.workunit.client.0.smithi071.stderr:ph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:20:39.608 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 141733920772 -lt 141733920772 2021-07-26T07:20:39.608 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:20:39.608 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-141733920770 2021-07-26T07:20:39.608 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:20:39.609 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:20:39.609 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-141733920770 2021-07-26T07:20:39.609 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:20:39.609 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=141733920770 2021-07-26T07:20:39.610 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 141733920770' 2021-07-26T07:20:39.610 INFO:tasks.workunit.client.0.smithi071.stderr:451177: waiting osd.3 seq 141733920770 2021-07-26T07:20:39.610 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:20:39.611 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 141733920772 -lt 141733920770 2021-07-26T07:20:39.612 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:20:39.612 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:39.612 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:39.613 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:20:39.613 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:20:39.613 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:20:39.613 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:20:39.614 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:20:39.614 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:20:39.614 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:20:39.614 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:20:39.615 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:20:39.615 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:20:39.615 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:39.615 INFO:tasks.workunit.client.0.smithi071.stderr:451177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:39.616 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:20:39.616 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:20:39.616 INFO:tasks.workunit.client.0.smithi071.stderr:451177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:20:39.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T07:20:39.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T07:20:39.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 451175 2021-07-26T07:20:39.617 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T07:20:39.617 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T07:20:39.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T07:20:39.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: unfound_erasure_coded: return_code=0 2021-07-26T07:20:39.618 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: unfound_erasure_coded: '[' 0 -ne 0 ']' 2021-07-26T07:20:39.618 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:896: unfound_erasure_coded: get_pg ecpool SOMETHING 2021-07-26T07:20:39.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T07:20:39.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T07:20:39.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T07:20:39.619 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:20:39.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:896: unfound_erasure_coded: local pg=1.0 2021-07-26T07:20:39.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: repair 1.0 2021-07-26T07:20:39.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=1.0 2021-07-26T07:20:39.898 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 1.0 2021-07-26T07:20:39.899 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:20:39.899 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:20:39.899 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:20:39.900 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:20:40.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T07:20:25.162687+0000 2021-07-26T07:20:40.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 1.0 2021-07-26T07:20:40.499 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 1.0s0 on osd.1 to repair 2021-07-26T07:20:40.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 1.0 2021-07-26T07:20:25.162687+0000 2021-07-26T07:20:40.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.0 2021-07-26T07:20:40.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:20:25.162687+0000 2021-07-26T07:20:40.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:20:40.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:20:40.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:20:40.512 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:20:40.512 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:20:40.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:20:40.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:20:40.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:20:40.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:20:25.162687+0000 '>' 2021-07-26T07:20:25.162687+0000 2021-07-26T07:20:40.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:20:41.771 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:20:41.772 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:20:41.773 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:20:41.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:20:41.774 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:20:41.775 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:20:41.775 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:20:42.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:20:25.162687+0000 '>' 2021-07-26T07:20:25.162687+0000 2021-07-26T07:20:42.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:20:43.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:20:43.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:20:43.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:20:43.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:20:43.047 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:20:43.047 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:20:43.047 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:20:43.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:20:25.162687+0000 '>' 2021-07-26T07:20:25.162687+0000 2021-07-26T07:20:43.312 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:20:44.313 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:20:44.314 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:20:44.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:20:44.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:20:44.315 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:20:44.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:20:44.316 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:20:44.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:20:25.162687+0000 '>' 2021-07-26T07:20:25.162687+0000 2021-07-26T07:20:44.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:20:45.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:20:45.587 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:20:45.588 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:20:45.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:20:45.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:20:45.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:20:45.589 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:20:45.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:20:25.162687+0000 '>' 2021-07-26T07:20:25.162687+0000 2021-07-26T07:20:45.882 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:20:46.883 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:20:46.884 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:20:46.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:20:46.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:20:46.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:20:46.885 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:20:46.886 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:20:47.154 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:20:25.162687+0000 '>' 2021-07-26T07:20:25.162687+0000 2021-07-26T07:20:47.155 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:20:48.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:20:48.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:20:48.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:20:48.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:20:48.157 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:20:48.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:20:48.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:20:48.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:20:41.772616+0000 '>' 2021-07-26T07:20:25.162687+0000 2021-07-26T07:20:48.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:20:48.425 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:902: unfound_erasure_coded: seq 1 60 2021-07-26T07:20:48.426 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:902: unfound_erasure_coded: for f in `seq 1 60` 2021-07-26T07:20:48.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:903: unfound_erasure_coded: ceph -s 2021-07-26T07:20:48.427 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:903: unfound_erasure_coded: grep '1/1 objects unfound' 2021-07-26T07:20:48.749 INFO:tasks.workunit.client.0.smithi071.stdout: 1/1 objects unfound (100.000%) 2021-07-26T07:20:48.750 INFO:tasks.workunit.client.0.smithi071.stdout: 1/1 objects unfound (100.000%) 2021-07-26T07:20:48.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:903: unfound_erasure_coded: break 2021-07-26T07:20:48.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:906: unfound_erasure_coded: ceph -s 2021-07-26T07:20:48.751 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:906: unfound_erasure_coded: grep '4 up' 2021-07-26T07:20:49.098 INFO:tasks.workunit.client.0.smithi071.stdout: osd: 4 osds: 4 up (since 14s), 4 in (since 29s) 2021-07-26T07:20:49.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: unfound_erasure_coded: ceph -s 2021-07-26T07:20:49.100 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: unfound_erasure_coded: grep '4 in' 2021-07-26T07:20:49.463 INFO:tasks.workunit.client.0.smithi071.stdout: osd: 4 osds: 4 up (since 15s), 4 in (since 29s) 2021-07-26T07:20:49.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:908: unfound_erasure_coded: ceph -s 2021-07-26T07:20:49.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:908: unfound_erasure_coded: grep '1/1 objects unfound' 2021-07-26T07:20:49.831 INFO:tasks.workunit.client.0.smithi071.stdout: 1/1 objects unfound (100.000%) 2021-07-26T07:20:49.831 INFO:tasks.workunit.client.0.smithi071.stdout: 1/1 objects unfound (100.000%) 2021-07-26T07:20:49.832 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T07:20:49.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:20:49.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:20:49.833 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:20:49.833 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:20:49.834 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:20:49.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:20:49.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:20:49.834 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:20:49.938 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:20:49.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:20:49.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:20:49.940 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:20:49.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:20:49.942 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:20:49.942 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:20:49.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:20:49.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:20:49.952 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:20:49.953 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:20:49.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:20:49.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:20:49.953 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:20:49.956 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:20:49.956 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:49.956 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:49.956 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:20:49.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:20:49.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:20:49.959 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2021-07-26T07:20:49.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2021-07-26T07:20:49.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2021-07-26T07:20:49.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2021-07-26T07:20:49.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:20:49.960 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:20:49.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:20:49.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:20:49.961 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:20:49.961 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:20:49.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:20:49.962 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:20:49.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:20:49.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:20:49.964 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:20:49.964 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:20:49.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:20:49.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:20:49.965 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:20:49.965 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:20:49.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:20:49.966 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:20:49.967 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:20:49.967 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:20:49.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:20:49.968 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:20:49.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:20:49.968 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:49.969 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:49.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:20:49.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:20:49.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:20:49.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2021-07-26T07:20:49.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:20:49.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:49.971 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:49.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.128899 2021-07-26T07:20:49.974 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:20:49.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:20:49.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:20:49.974 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2021-07-26T07:20:49.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_unfound_erasure_coded_overwrites td/osd-scrub-repair 2021-07-26T07:20:49.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: TEST_unfound_erasure_coded_overwrites: '[' true = true ']' 2021-07-26T07:20:49.975 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: TEST_unfound_erasure_coded_overwrites: unfound_erasure_coded td/osd-scrub-repair true 2021-07-26T07:20:49.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: local dir=td/osd-scrub-repair 2021-07-26T07:20:49.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: local allow_overwrites=true 2021-07-26T07:20:49.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: local poolname=ecpool 2021-07-26T07:20:49.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: local payload=ABCDEF 2021-07-26T07:20:49.976 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:862: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2021-07-26T07:20:49.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2021-07-26T07:20:49.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:20:49.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:20:49.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:20:49.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2021-07-26T07:20:49.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2021-07-26T07:20:50.355 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:20:50.355 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:50.355 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:50.356 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:50.356 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:50.356 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:50.356 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:50.357 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:20:50.606 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:20:50.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:20:50.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:20:50.607 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:20:50.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:20:50.608 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:20:50.609 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:20:50.609 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:20:50.610 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:20:50.610 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:20:50.610 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:50.610 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:50.611 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:20:50.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:20:50.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get fsid 2021-07-26T07:20:50.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:20:50.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:20:50.706 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:20:50.707 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:20:50.707 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:20:50.707 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:20:50.707 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:20:50.707 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:20:50.708 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:20:50.708 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:50.708 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:50.708 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.128899/ceph-mon.a.asok 2021-07-26T07:20:50.709 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:20:50.709 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.128899/ceph-mon.a.asok config get mon_host 2021-07-26T07:20:50.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2021-07-26T07:20:50.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2021-07-26T07:20:50.803 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:20:50.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:20:50.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:20:50.804 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2021-07-26T07:20:50.805 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:20:51.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:20:51.636 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:51.637 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:51.637 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:51.637 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:51.637 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:51.638 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:51.639 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:20:51.639 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:20:51.659 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: unfound_erasure_coded: seq 0 3 2021-07-26T07:20:51.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: unfound_erasure_coded: for id in $(seq 0 3) 2021-07-26T07:20:51.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: '[' true = true ']' 2021-07-26T07:20:51.666 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:866: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2021-07-26T07:20:51.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:20:51.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:20:51.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:20:51.667 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:20:51.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:20:51.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:20:51.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:20:51.668 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:20:51.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:20:51.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:20:51.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:20:51.669 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:20:51.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:20:51.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:20:51.670 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:20:51.670 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:20:51.672 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:51.672 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:51.673 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:51.673 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:51.673 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:20:51.673 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:51.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:20:51.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:20:51.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:20:51.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:20:51.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:20:51.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:20:51.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:20:51.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:20:51.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:51.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:20:51.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:20:51.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:20:51.677 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:20:51.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=caa4d9ef-a11a-4375-a2d1-fdcf99acc624 2021-07-26T07:20:51.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 caa4d9ef-a11a-4375-a2d1-fdcf99acc624' 2021-07-26T07:20:51.678 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 caa4d9ef-a11a-4375-a2d1-fdcf99acc624 2021-07-26T07:20:51.678 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:20:51.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTYv5geUtzKBAAewowckH6bveAHCSSdDgtmA== 2021-07-26T07:20:51.679 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTYv5geUtzKBAAewowckH6bveAHCSSdDgtmA=="}' 2021-07-26T07:20:51.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new caa4d9ef-a11a-4375-a2d1-fdcf99acc624 -i td/osd-scrub-repair/0/new.json 2021-07-26T07:20:51.982 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:20:51.993 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2021-07-26T07:20:51.994 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBTYv5geUtzKBAAewowckH6bveAHCSSdDgtmA== --osd-uuid caa4d9ef-a11a-4375-a2d1-fdcf99acc624 2021-07-26T07:20:52.018 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:52.017+0000 7f561851c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:52.020 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:52.019+0000 7f561851c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:52.020 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:52.020+0000 7f561851c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:52.021 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:52.020+0000 7f561851c0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:20:52.021 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:52.020+0000 7f561851c0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:20:52.022 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:52.020+0000 7f561851c0c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2021-07-26T07:20:52.073 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:52.072+0000 7f561851c0c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2021-07-26T07:20:57.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2021-07-26T07:20:57.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:20:57.694 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:20:57.695 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:20:57.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:20:58.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:20:58.033 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:20:58.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:58.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:20:58.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:20:58.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:20:58.055 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:58.054+0000 7f87ec8a20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:58.056 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:58.056+0000 7f87ec8a20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:58.057 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:58.057+0000 7f87ec8a20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:58.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:20:58.343 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:20:58.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:20:58.344 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:20:58.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:20:58.345 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:58.346 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:20:58.346 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:20:58.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:58.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:20:58.574 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:20:58.574+0000 7f87ec8a20c0 -1 Falling back to public interface 2021-07-26T07:20:58.641 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:59.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:59.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:59.643 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:20:59.643 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:20:59.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:59.644 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:20:59.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:00.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:00.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:00.919 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:21:00.920 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:21:00.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:00.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:21:00.922 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:00.922+0000 7f87ec8a20c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:21:01.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:02.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:02.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:02.196 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:21:02.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:21:02.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:02.198 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:21:02.474 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/454650,v1:127.0.0.1:6803/454650] [v2:127.0.0.1:6804/454650,v1:127.0.0.1:6805/454650] exists,up caa4d9ef-a11a-4375-a2d1-fdcf99acc624 2021-07-26T07:21:02.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:21:02.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:21:02.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:21:02.483 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: unfound_erasure_coded: for id in $(seq 0 3) 2021-07-26T07:21:02.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: '[' true = true ']' 2021-07-26T07:21:02.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:866: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2021-07-26T07:21:02.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:21:02.484 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:21:02.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:21:02.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:21:02.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2021-07-26T07:21:02.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:21:02.485 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:21:02.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:21:02.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:21:02.486 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2021-07-26T07:21:02.495 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2021-07-26T07:21:02.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:21:02.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:21:02.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:21:02.496 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:21:02.496 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:21:02.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:21:02.497 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:21:02.497 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:21:02.497 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:21:02.498 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:21:02.498 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:21:02.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:21:02.498 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:21:02.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:21:02.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:21:02.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:21:02.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:21:02.499 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:21:02.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:21:02.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:02.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:21:02.500 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:21:02.501 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2021-07-26T07:21:02.504 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:21:02.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0d26e1dd-3b37-4b01-af83-1e5f7fe810c9 2021-07-26T07:21:02.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0d26e1dd-3b37-4b01-af83-1e5f7fe810c9' 2021-07-26T07:21:02.505 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 0d26e1dd-3b37-4b01-af83-1e5f7fe810c9 2021-07-26T07:21:02.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:21:02.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBeYv5gN5laHRAAmkPSYr4RZHzRofzD3EgERw== 2021-07-26T07:21:02.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBeYv5gN5laHRAAmkPSYr4RZHzRofzD3EgERw=="}' 2021-07-26T07:21:02.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0d26e1dd-3b37-4b01-af83-1e5f7fe810c9 -i td/osd-scrub-repair/1/new.json 2021-07-26T07:21:02.882 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:21:02.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2021-07-26T07:21:02.895 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBeYv5gN5laHRAAmkPSYr4RZHzRofzD3EgERw== --osd-uuid 0d26e1dd-3b37-4b01-af83-1e5f7fe810c9 2021-07-26T07:21:02.918 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:02.917+0000 7f03f6c780c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:02.920 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:02.919+0000 7f03f6c780c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:02.920 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:02.920+0000 7f03f6c780c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:02.921 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:02.920+0000 7f03f6c780c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:21:02.922 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:02.920+0000 7f03f6c780c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:21:02.923 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:02.920+0000 7f03f6c780c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2021-07-26T07:21:02.991 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:02.990+0000 7f03f6c780c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2021-07-26T07:21:08.437 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2021-07-26T07:21:08.438 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:21:08.438 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T07:21:08.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:21:08.439 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:21:08.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:21:08.817 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T07:21:08.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:08.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:21:08.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:21:08.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:21:08.837 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:08.837+0000 7fd770e770c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:08.839 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:08.838+0000 7fd770e770c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:08.840 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:08.839+0000 7fd770e770c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:09.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:21:09.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:21:09.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:21:09.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:21:09.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:21:09.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:09.087 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:21:09.088 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:21:09.088 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:09.089 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:21:09.362 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:09.360+0000 7fd770e770c0 -1 Falling back to public interface 2021-07-26T07:21:09.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:10.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:10.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:10.398 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:21:10.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:21:10.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:10.399 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:21:10.676 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:11.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:11.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:11.679 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:21:11.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:21:11.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:11.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:21:11.700 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:11.699+0000 7fd770e770c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:21:12.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:13.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:13.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:13.022 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:21:13.022 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:21:13.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:13.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:21:13.287 INFO:tasks.workunit.client.0.smithi071.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/456049,v1:127.0.0.1:6811/456049] [v2:127.0.0.1:6812/456049,v1:127.0.0.1:6813/456049] exists,up 0d26e1dd-3b37-4b01-af83-1e5f7fe810c9 2021-07-26T07:21:13.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:21:13.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:21:13.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:21:13.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: unfound_erasure_coded: for id in $(seq 0 3) 2021-07-26T07:21:13.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: '[' true = true ']' 2021-07-26T07:21:13.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:866: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2021-07-26T07:21:13.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:21:13.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:21:13.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T07:21:13.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:21:13.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:21:13.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:21:13.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:21:13.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:21:13.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:21:13.297 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T07:21:13.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T07:21:13.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:21:13.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:21:13.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:21:13.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:21:13.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:21:13.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:21:13.299 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:21:13.300 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:21:13.300 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:21:13.300 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:21:13.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:21:13.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:21:13.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:21:13.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:21:13.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:21:13.301 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:21:13.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:21:13.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:21:13.302 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:21:13.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:13.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:21:13.303 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:21:13.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T07:21:13.304 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:21:13.304 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e5d335fa-28d8-4dc2-b86b-256684f4ffec 2021-07-26T07:21:13.305 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e5d335fa-28d8-4dc2-b86b-256684f4ffec' 2021-07-26T07:21:13.306 INFO:tasks.workunit.client.0.smithi071.stdout:add osd2 e5d335fa-28d8-4dc2-b86b-256684f4ffec 2021-07-26T07:21:13.307 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:21:13.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpYv5g7XYoEhAAXW6/k5p5oGHLvWi/BkfBBw== 2021-07-26T07:21:13.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpYv5g7XYoEhAAXW6/k5p5oGHLvWi/BkfBBw=="}' 2021-07-26T07:21:13.307 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e5d335fa-28d8-4dc2-b86b-256684f4ffec -i td/osd-scrub-repair/2/new.json 2021-07-26T07:21:13.651 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:21:13.663 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2021-07-26T07:21:13.664 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBpYv5g7XYoEhAAXW6/k5p5oGHLvWi/BkfBBw== --osd-uuid e5d335fa-28d8-4dc2-b86b-256684f4ffec 2021-07-26T07:21:13.687 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:13.687+0000 7ff54c6630c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:13.717 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:13.717+0000 7ff54c6630c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:13.718 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:13.717+0000 7ff54c6630c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:13.719 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:13.718+0000 7ff54c6630c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T07:21:13.719 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:13.718+0000 7ff54c6630c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T07:21:13.719 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:13.718+0000 7ff54c6630c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2021-07-26T07:21:13.793 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:13.792+0000 7ff54c6630c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2021-07-26T07:21:19.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2021-07-26T07:21:19.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:21:19.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T07:21:19.601 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd2 key to auth repository 2021-07-26T07:21:19.601 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:21:19.939 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T07:21:19.939 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.2 2021-07-26T07:21:19.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:21:19.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:19.940 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:21:19.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:21:19.961 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:19.960+0000 7facdbd0e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:19.962 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:19.962+0000 7facdbd0e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:19.963 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:19.963+0000 7facdbd0e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:20.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T07:21:20.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:21:20.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:21:20.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:21:20.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:21:20.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:20.251 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:21:20.252 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:21:20.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:20.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:21:20.478 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:20.477+0000 7facdbd0e0c0 -1 Falling back to public interface 2021-07-26T07:21:20.583 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:21.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:21.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:21.585 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:21:21.585 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:21:21.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:21.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:21:21.861 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:22.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:22.863 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:22.864 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:21:22.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:21:22.864 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:22.865 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:21:22.944 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:22.943+0000 7facdbd0e0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T07:21:23.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:24.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:24.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:24.151 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:21:24.152 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:21:24.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:24.152 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:21:24.458 INFO:tasks.workunit.client.0.smithi071.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/457456,v1:127.0.0.1:6819/457456] [v2:127.0.0.1:6820/457456,v1:127.0.0.1:6821/457456] exists,up e5d335fa-28d8-4dc2-b86b-256684f4ffec 2021-07-26T07:21:24.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:21:24.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:21:24.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:21:24.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: unfound_erasure_coded: for id in $(seq 0 3) 2021-07-26T07:21:24.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: '[' true = true ']' 2021-07-26T07:21:24.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:866: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2021-07-26T07:21:24.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2021-07-26T07:21:24.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:21:24.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T07:21:24.465 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:21:24.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T07:21:24.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:21:24.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:21:24.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:21:24.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:21:24.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T07:21:24.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T07:21:24.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:21:24.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:21:24.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:21:24.468 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:21:24.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:21:24.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:21:24.468 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:21:24.469 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:21:24.469 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:21:24.469 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:21:24.469 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:21:24.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:21:24.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:21:24.470 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:21:24.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:21:24.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:21:24.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:21:24.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:21:24.471 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:21:24.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:24.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:21:24.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:21:24.472 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T07:21:24.472 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:21:24.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f8feb347-a7d8-4955-b91c-9bfed665bde7 2021-07-26T07:21:24.473 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 f8feb347-a7d8-4955-b91c-9bfed665bde7' 2021-07-26T07:21:24.473 INFO:tasks.workunit.client.0.smithi071.stdout:add osd3 f8feb347-a7d8-4955-b91c-9bfed665bde7 2021-07-26T07:21:24.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:21:24.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0Yv5g+hRkHBAA7JHv3rM4avzdki+V4U821g== 2021-07-26T07:21:24.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0Yv5g+hRkHBAA7JHv3rM4avzdki+V4U821g=="}' 2021-07-26T07:21:24.477 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f8feb347-a7d8-4955-b91c-9bfed665bde7 -i td/osd-scrub-repair/3/new.json 2021-07-26T07:21:24.881 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:21:24.893 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2021-07-26T07:21:24.894 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB0Yv5g+hRkHBAA7JHv3rM4avzdki+V4U821g== --osd-uuid f8feb347-a7d8-4955-b91c-9bfed665bde7 2021-07-26T07:21:24.918 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:24.917+0000 7ff0c5b240c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:24.948 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:24.947+0000 7ff0c5b240c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:24.949 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:24.948+0000 7ff0c5b240c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:24.951 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:24.949+0000 7ff0c5b240c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2021-07-26T07:21:24.951 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:24.949+0000 7ff0c5b240c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2021-07-26T07:21:24.952 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:24.949+0000 7ff0c5b240c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2021-07-26T07:21:25.036 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:25.035+0000 7ff0c5b240c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2021-07-26T07:21:30.690 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2021-07-26T07:21:30.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:21:30.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T07:21:30.692 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd3 key to auth repository 2021-07-26T07:21:30.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:21:31.051 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.3 2021-07-26T07:21:31.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T07:21:31.052 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:21:31.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:31.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:21:31.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:21:31.073 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:31.071+0000 7fedfe2e40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:31.074 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:31.073+0000 7fedfe2e40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:31.076 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:31.074+0000 7fedfe2e40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:31.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T07:21:31.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:21:31.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T07:21:31.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:21:31.361 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:21:31.361 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:21:31.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:31.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:21:31.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:31.364 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:21:31.593 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:31.591+0000 7fedfe2e40c0 -1 Falling back to public interface 2021-07-26T07:21:31.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:32.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:32.673 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:32.673 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:21:32.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:21:32.674 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:32.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:21:32.988 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:33.928 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:21:33.926+0000 7fedfe2e40c0 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T07:21:33.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:33.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:33.990 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:21:33.991 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:21:33.991 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:33.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:21:34.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:35.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:35.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:35.329 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:21:35.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:21:35.330 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:35.331 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:21:35.592 INFO:tasks.workunit.client.0.smithi071.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/458855,v1:127.0.0.1:6827/458855] [v2:127.0.0.1:6828/458855,v1:127.0.0.1:6829/458855] exists,up f8feb347-a7d8-4955-b91c-9bfed665bde7 2021-07-26T07:21:35.593 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:21:35.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:21:35.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:21:35.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:872: unfound_erasure_coded: create_ec_pool ecpool true k=2 m=2 2021-07-26T07:21:35.594 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=ecpool 2021-07-26T07:21:35.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T07:21:35.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=true 2021-07-26T07:21:35.595 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T07:21:35.596 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2021-07-26T07:21:36.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2021-07-26T07:21:36.051 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2021-07-26T07:21:36.730 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'ecpool' created 2021-07-26T07:21:36.740 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:21:37.741 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' true = true ']' 2021-07-26T07:21:37.742 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2021-07-26T07:21:38.229 INFO:tasks.workunit.client.0.smithi071.stderr:set pool 1 allow_ec_overwrites to true 2021-07-26T07:21:38.239 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T07:21:38.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:21:38.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:21:38.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:21:38.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:21:38.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:21:38.241 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:21:38.241 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:21:38.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:21:38.241 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:21:38.242 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:21:38.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:21:38.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:21:38.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:21:38.452 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:21:38.452 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:21:38.675 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:21:38.676 INFO:tasks.workunit.client.0.smithi071.stderr:1 2021-07-26T07:21:38.676 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:21:38.676 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T07:21:38.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:21:38.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:38.677 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:21:38.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836489 2021-07-26T07:21:38.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836489 2021-07-26T07:21:38.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489' 2021-07-26T07:21:38.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:38.826 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:21:38.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672968 2021-07-26T07:21:38.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672968 2021-07-26T07:21:38.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2021-07-26T07:21:38.944 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:38.944 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:21:39.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509444 2021-07-26T07:21:39.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509444 2021-07-26T07:21:39.061 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509444' 2021-07-26T07:21:39.062 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:39.062 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:21:39.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345923 2021-07-26T07:21:39.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345923 2021-07-26T07:21:39.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509444 3-85899345923' 2021-07-26T07:21:39.180 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:39.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836489 2021-07-26T07:21:39.180 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:39.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:21:39.182 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836489 2021-07-26T07:21:39.182 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:39.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836489 2021-07-26T07:21:39.183 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2021-07-26T07:21:39.183 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836489 2021-07-26T07:21:39.184 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:21:39.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836488 -lt 21474836489 2021-07-26T07:21:39.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:21:40.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:21:40.449 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:21:40.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836489 2021-07-26T07:21:40.729 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:40.730 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672968 2021-07-26T07:21:40.730 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:40.731 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:21:40.731 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672968 2021-07-26T07:21:40.732 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:40.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672968 2021-07-26T07:21:40.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-07-26T07:21:40.733 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.1 seq 42949672968 2021-07-26T07:21:40.733 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:21:41.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672968 -lt 42949672968 2021-07-26T07:21:41.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:41.034 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509444 2021-07-26T07:21:41.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:41.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:21:41.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509444 2021-07-26T07:21:41.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:41.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509444 2021-07-26T07:21:41.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-07-26T07:21:41.037 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.2 seq 64424509444 2021-07-26T07:21:41.038 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:21:41.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509445 -lt 64424509444 2021-07-26T07:21:41.334 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:41.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345923 2021-07-26T07:21:41.335 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:41.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:21:41.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345923 2021-07-26T07:21:41.336 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:41.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345923 2021-07-26T07:21:41.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345923' 2021-07-26T07:21:41.338 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.3 seq 85899345923 2021-07-26T07:21:41.338 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:21:41.642 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345923 -lt 85899345923 2021-07-26T07:21:41.643 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:21:41.643 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:21:41.644 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:21:42.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:21:42.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:21:42.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:21:42.139 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:21:42.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:21:42.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:21:42.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:21:42.140 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:21:42.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:21:42.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:21:42.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:21:42.441 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:21:42.767 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:21:42.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:21:42.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:21:42.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T07:21:42.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:874: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2021-07-26T07:21:42.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2021-07-26T07:21:42.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2021-07-26T07:21:42.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2021-07-26T07:21:42.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2021-07-26T07:21:42.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2021-07-26T07:21:42.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2021-07-26T07:21:43.199 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:21:43.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2021-07-26T07:21:43.718 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:21:43.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2021-07-26T07:21:43.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2021-07-26T07:21:43.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2021-07-26T07:21:44.045 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: get_primary ecpool SOMETHING 2021-07-26T07:21:44.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=ecpool 2021-07-26T07:21:44.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=SOMETHING 2021-07-26T07:21:44.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map ecpool SOMETHING 2021-07-26T07:21:44.046 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:21:44.315 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: local primary=1 2021-07-26T07:21:44.316 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2021-07-26T07:21:44.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: get_osds ecpool SOMETHING 2021-07-26T07:21:44.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: sed -e s/1// 2021-07-26T07:21:44.317 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=ecpool 2021-07-26T07:21:44.318 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=SOMETHING 2021-07-26T07:21:44.318 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map ecpool SOMETHING 2021-07-26T07:21:44.318 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:21:44.604 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=1 2021-07-26T07:21:44.605 INFO:tasks.workunit.client.0.smithi071.stderr:0 2021-07-26T07:21:44.605 INFO:tasks.workunit.client.0.smithi071.stderr:2 2021-07-26T07:21:44.605 INFO:tasks.workunit.client.0.smithi071.stderr:3' 2021-07-26T07:21:44.605 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 1 0 2 3 2021-07-26T07:21:44.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: local -a osds 2021-07-26T07:21:44.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: local not_primary_first=0 2021-07-26T07:21:44.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: local not_primary_second=2 2021-07-26T07:21:44.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: local not_primary_third=3 2021-07-26T07:21:44.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:885: unfound_erasure_coded: pids= 2021-07-26T07:21:44.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:886: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T07:21:44.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T07:21:44.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T07:21:44.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 460039"' 2021-07-26T07:21:44.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 460039' 2021-07-26T07:21:44.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T07:21:44.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T07:21:44.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T07:21:44.612 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 460040"' 2021-07-26T07:21:44.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 460040' 2021-07-26T07:21:44.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T07:21:44.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: run_in_background: local pid_variable=pids 2021-07-26T07:21:44.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T07:21:44.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: run_in_background: shift 2021-07-26T07:21:44.613 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/460042: /' 2021-07-26T07:21:44.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T07:21:44.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: eval 'pids+=" 460044"' 2021-07-26T07:21:44.614 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: run_in_background: pids+=' 460044' 2021-07-26T07:21:44.614 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/460045: /' 2021-07-26T07:21:44.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:889: unfound_erasure_coded: wait_background pids 2021-07-26T07:21:44.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: wait_background: pids=' 460039 460040 460044' 2021-07-26T07:21:44.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: wait_background: return_code=0 2021-07-26T07:21:44.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T07:21:44.615 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 460039 2021-07-26T07:21:44.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T07:21:44.616 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: sed 's/^/460048: /' 2021-07-26T07:21:47.446 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:21:47.447 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:21:47.447 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=3 2021-07-26T07:21:47.447 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:21:47.447 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T07:21:47.448 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:21:47.448 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:21:47.448 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=3 2021-07-26T07:21:47.448 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:21:47.448 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2021-07-26T07:21:47.449 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:21:47.449 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:21:47.449 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:21:47.449 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:21:47.450 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:21:47.450 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:21:47.450 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2021-07-26T07:21:47.450 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:21:47.450 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:21:47.451 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=3 2021-07-26T07:21:47.451 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:21:47.451 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2021-07-26T07:21:47.451 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2021-07-26T07:21:47.452 INFO:tasks.workunit.client.0.smithi071.stderr:460048: Error getting attr on : 1.0s3_head,3#1:00000000::::head#, (61) No data available 2021-07-26T07:21:47.452 INFO:tasks.workunit.client.0.smithi071.stderr:460048: remove 3#1:eb822e21:::SOMETHING:head# 2021-07-26T07:21:47.452 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2021-07-26T07:21:47.452 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:21:47.453 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:21:47.453 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T07:21:47.453 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:21:47.453 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/3 2021-07-26T07:21:47.453 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:21:47.454 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:21:47.454 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:21:47.455 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:21:47.455 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2021-07-26T07:21:47.455 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2021-07-26T07:21:47.455 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:21:47.456 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:21:47.456 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:21:47.456 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:21:47.456 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:21:47.456 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:21:47.457 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:21:47.457 INFO:tasks.workunit.client.0.smithi071.stderr:460048: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:21:47.457 INFO:tasks.workunit.client.0.smithi071.stderr:460048: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:21:47.457 INFO:tasks.workunit.client.0.smithi071.stderr:460048: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:21:47.457 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:21:47.458 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:21:47.458 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:21:47.458 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:21:47.458 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:21:47.459 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:21:47.459 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:21:47.459 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:47.459 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:21:47.459 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:21:47.460 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/3 2021-07-26T07:21:47.460 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T07:21:47.460 INFO:tasks.workunit.client.0.smithi071.stderr:460048: start osd.3 2021-07-26T07:21:47.479 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-ex460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:21:47.480 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:21:47.480 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=0 2021-07-26T07:21:47.480 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:21:47.480 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T07:21:47.481 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:21:47.481 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:21:47.481 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=0 2021-07-26T07:21:47.481 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:21:47.481 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2021-07-26T07:21:47.482 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:21:47.482 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:21:47.482 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:21:47.482 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:21:47.483 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:21:47.483 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:21:47.483 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2021-07-26T07:21:47.483 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:21:47.483 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:21:47.484 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T07:21:47.484 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:21:47.484 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:21:47.484 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2021-07-26T07:21:47.485 INFO:tasks.workunit.client.0.smithi071.stderr:460042: Error getting attr on : 1.0s1_head,1#1:00000000::::head#, (61) No data available 2021-07-26T07:21:47.485 INFO:tasks.workunit.client.0.smithi071.stderr:460042: remove 1#1:eb822e21:::SOMETHING:head# 2021-07-26T07:21:47.485 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2021-07-26T07:21:47.485 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:21:47.486 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:21:47.486 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:21:47.486 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:21:47.486 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/0 2021-07-26T07:21:47.486 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:21:47.487 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ra460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: local dir=td/osd-scrub-repair 2021-07-26T07:21:47.487 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: shift 2021-07-26T07:21:47.487 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: local id=2 2021-07-26T07:21:47.488 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: objectstore_tool: shift 2021-07-26T07:21:47.488 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T07:21:47.488 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2021-07-26T07:21:47.488 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: shift 2021-07-26T07:21:47.488 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: local id=2 2021-07-26T07:21:47.489 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: shift 2021-07-26T07:21:47.489 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1216: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2021-07-26T07:21:47.490 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:21:47.490 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:21:47.490 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:21:47.490 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:21:47.491 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:21:47.491 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:21:47.491 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1218: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2021-07-26T07:21:47.491 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2021-07-26T07:21:47.491 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T07:21:47.492 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=2 2021-07-26T07:21:47.492 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T07:21:47.492 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:21:47.492 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2021-07-26T07:21:47.493 INFO:tasks.workunit.client.0.smithi071.stderr:460045: Error getting attr on : 1.0s2_head,2#1:00000000::::head#, (61) No data available 2021-07-26T07:21:47.493 INFO:tasks.workunit.client.0.smithi071.stderr:460045: remove 2#1:eb822e21:::SOMETHING:head# 2021-07-26T07:21:47.493 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1219: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2021-07-26T07:21:47.493 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-repair 2021-07-26T07:21:47.493 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:21:47.494 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T07:21:47.494 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:21:47.494 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-repair/2 2021-07-26T07:21:47.494 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2021-07-26T07:21:47.495 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:21:47.495 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:21:47.496 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:21:47.496 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2021-07-26T07:21:47.496 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2021-07-26T07:21:47.496 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:21:47.497 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:21:47.497 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:21:47.497 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:21:47.497 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:21:47.497 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:21:47.498 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:21:47.498 INFO:tasks.workunit.client.0.smithi071.stderr:460045: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:21:47.498 INFO:tasks.workunit.client.0.smithi071.stderr:460045: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:21:47.498 INFO:tasks.workunit.client.0.smithi071.stderr:460045: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:21:47.499 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:21:47.499 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:21:47.499 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:21:47.499 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:21:47.500 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:21:47.500 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:21:47.500 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:21:47.500 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:47.500 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:21:47.501 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:21:47.501 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/2 2021-07-26T07:21:47.501 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T07:21:47.501 INFO:tasks.workunit.client.0.smithi071.stderr:460045: start osd.2 2021-07-26T07:21:47.502 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-extio=.99' 2021-07-26T07:21:47.502 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:21:47.502 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:21:47.502 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2021-07-26T07:21:47.502 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2021-07-26T07:21:47.503 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:21:47.503 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:21:47.503 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:21:47.503 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2021-07-26T07:21:47.504 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:21:47.504 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:21:47.504 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:21:47.504 INFO:tasks.workunit.client.0.smithi071.stderr:460042: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:21:47.504 INFO:tasks.workunit.client.0.smithi071.stderr:460042: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:21:47.505 INFO:tasks.workunit.client.0.smithi071.stderr:460042: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:21:47.505 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:21:47.505 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' 2021-07-26T07:21:47.505 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:21:47.506 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2021-07-26T07:21:47.506 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2021-07-26T07:21:47.506 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:21:47.506 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:21:47.506 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:47.507 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:21:47.507 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:21:47.507 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-repair/0 2021-07-26T07:21:47.507 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:21:47.508 INFO:tasks.workunit.client.0.smithi071.stderr:460042: start osd.0 2021-07-26T07:21:52.073 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=962d35b9-e269-4a9b-b9ff-14eb6d9675c0 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.128899/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:52.074 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/2/whoami 2021-07-26T07:21:52.074 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T07:21:52.074 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:21:52.074 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:21:52.075 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:21:52.075 INFO:tasks.workunit.client.0.smithi071.stderr:460045: 2021-07-26T07:21:47.506+0000 7f28149a20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:52.075 INFO:tasks.workunit.client.0.smithi071.stderr:460045: 2021-07-26T07:21:47.509+0000 7f28149a20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:52.075 INFO:tasks.workunit.client.0.smithi071.stderr:460045: 2021-07-26T07:21:47.510+0000 7f28149a20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:52.076 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T07:21:52.076 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:21:52.076 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:21:52.076 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:21:52.076 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:21:52.077 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:52.077 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:21:52.077 INFO:tasks.workunit.client.0.smithi071.stderr:460045: 0 2021-07-26T07:21:52.077 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:52.077 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:21:52.078 INFO:tasks.workunit.client.0.smithi071.stderr:460045: 2021-07-26T07:21:48.033+0000 7f28149a20c0 -1 Falling back to public interface 2021-07-26T07:21:52.078 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:52.078 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:52.078 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:52.079 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:21:52.079 INFO:tasks.workunit.client.0.smithi071.stderr:460045: 1 2021-07-26T07:21:52.079 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:52.079 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:21:52.079 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:52.080 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:52.080 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:52.080 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:21:52.080 INFO:tasks.workunit.client.0.smithi071.stderr:460045: 2 2021-07-26T07:21:52.080 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:52.081 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:21:52.081 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:52.081 INFO:tasks.workunit.client.0.smithi071.stderr:460045: 2021-07-26T07:21:51.091+0000 7f28149a20c0 -1 osd.2 29 log_to_monitors {default=true} 2021-07-26T07:21:52.081 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:52.082 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:52.082 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:21:52.082 INFO:tasks.workunit.client.0.smithi071.stderr:460045: 3 2021-07-26T07:21:52.082 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:52.082 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:21:52.083 INFO:tasks.workunit.client.0.smithi071.stderr:460045:perimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:52.083 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/0/whoami 2021-07-26T07:21:52.083 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:21:52.083 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:21:52.083 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:21:52.084 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:21:52.084 INFO:tasks.workunit.client.0.smithi071.stderr:460042: 2021-07-26T07:21:47.520+0000 7f507e6590c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:52.084 INFO:tasks.workunit.client.0.smithi071.stderr:460042: 2021-07-26T07:21:47.523+0000 7f507e6590c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:52.084 INFO:tasks.workunit.client.0.smithi071.stderr:460042: 2021-07-26T07:21:47.524+0000 7f507e6590c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:52.085 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:21:52.085 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:21:52.085 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:21:52.085 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:21:52.085 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:21:52.086 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:52.086 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:21:52.086 INFO:tasks.workunit.client.0.smithi071.stderr:460042: 0 2021-07-26T07:21:52.086 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:52.086 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:21:52.087 INFO:tasks.workunit.client.0.smithi071.stderr:460042: 2021-07-26T07:21:48.050+0000 7f507e6590c0 -1 Falling back to public interface 2021-07-26T07:21:52.087 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:52.087 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:52.087 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:52.087 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:21:52.088 INFO:tasks.workunit.client.0.smithi071.stderr:460042: 1 2021-07-26T07:21:52.088 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:52.088 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:21:52.088 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:52.089 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:52.089 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:52.089 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:21:52.089 INFO:tasks.workunit.client.0.smithi071.stderr:460042: 2 2021-07-26T07:21:52.089 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:52.090 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:21:52.090 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:52.090 INFO:tasks.workunit.client.0.smithi071.stderr:460042: 2021-07-26T07:21:51.142+0000 7f507e6590c0 -1 osd.0 29 log_to_monitors {default=true} 2021-07-26T07:21:52.090 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:52.090 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:52.091 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:21:52.091 INFO:tasks.workunit.client.0.smithi071.stderr:460042: 3 2021-07-26T07:21:52.091 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:52.091 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:21:52.092 INFO:tasks.workunit.client.0.smithi071.stderr:460042:perimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:52.092 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-repair/3/whoami 2021-07-26T07:21:52.092 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T07:21:52.092 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:21:52.092 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:21:52.093 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:21:52.093 INFO:tasks.workunit.client.0.smithi071.stderr:460048: 2021-07-26T07:21:47.473+0000 7f8f00f9b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:52.093 INFO:tasks.workunit.client.0.smithi071.stderr:460048: 2021-07-26T07:21:47.476+0000 7f8f00f9b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:52.093 INFO:tasks.workunit.client.0.smithi071.stderr:460048: 2021-07-26T07:21:47.477+0000 7f8f00f9b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:52.093 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T07:21:52.094 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:21:52.094 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T07:21:52.094 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:21:52.094 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:21:52.094 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:52.095 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:21:52.095 INFO:tasks.workunit.client.0.smithi071.stderr:460048: 0 2021-07-26T07:21:52.095 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:52.095 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:21:52.096 INFO:tasks.workunit.client.0.smithi071.stderr:460048: 2021-07-26T07:21:48.007+0000 7f8f00f9b0c0 -1 Falling back to public interface 2021-07-26T07:21:52.096 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:52.096 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:52.096 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:52.096 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:21:52.097 INFO:tasks.workunit.client.0.smithi071.stderr:460048: 1 2021-07-26T07:21:52.097 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:52.097 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:21:52.097 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:52.097 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:52.098 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:52.098 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:21:52.098 INFO:tasks.workunit.client.0.smithi071.stderr:460048: 2 2021-07-26T07:21:52.098 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:52.098 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:21:52.099 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:52.099 INFO:tasks.workunit.client.0.smithi071.stderr:460048: 2021-07-26T07:21:50.924+0000 7f8f00f9b0c0 -1 osd.3 29 log_to_monitors {default=true} 2021-07-26T07:21:52.099 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:52.099 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:52.100 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:21:52.100 INFO:tasks.workunit.client.0.smithi071.stderr:460048: 3 2021-07-26T07:21:52.100 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:52.100 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:21:52.744 INFO:tasks.workunit.client.0.smithi071.stderr:460048: osd.3 up in weight 1 up_from 34 up_thru 0 down_at 30 last_clean_interval [20,29) [v2:127.0.0.1:6802/461243,v1:127.0.0.1:6803/461243] [v2:127.0.0.1:6804/461243,v1:127.0.0.1:6805/461243] exists,up f8feb347-a7d8-4955-b91c-9bfed665bde7 2021-07-26T07:21:52.744 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:21:52.744 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:21:52.745 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:21:52.745 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:21:52.745 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:21:52.745 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:21:52.746 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:21:52.746 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:21:52.746 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:21:52.746 INFO:tasks.workunit.client.0.smithi071.stderr:460048: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:21:52.746 INFO:tasks.workunit.client.0.smithi071.stderr:460048: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:21:52.747 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:21:52.747 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:21:52.747 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:21:52.747 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:21:52.748 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:21:52.748 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:21:52.748 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:21:52.748 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:21:52.748 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:21:52.749 INFO:tasks.workunit.client.0.smithi071.stderr:460048: 1 2021-07-26T07:21:52.749 INFO:tasks.workunit.client.0.smithi071.stderr:460048: 2 2021-07-26T07:21:52.749 INFO:tasks.workunit.client.0.smithi071.stderr:460048: 3' 2021-07-26T07:21:52.749 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:21:52.749 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:52.750 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:21:52.750 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888068 2021-07-26T07:21:52.750 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888068 2021-07-26T07:21:52.750 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888068' 2021-07-26T07:21:52.751 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:52.751 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:21:52.751 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672971 2021-07-26T07:21:52.751 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672971 2021-07-26T07:21:52.751 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888068 1-42949672971' 2021-07-26T07:21:52.752 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:52.752 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: f osd.0 up in weight 1 up_from 34 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6826/461293,v1:127.0.0.1:6827/461293] [v2:127.0.0.1:6828/461293,v1:127.0.0.1:6829/461293] exists,up caa4d9ef-a11a-4375-a2d1-fdcf99acc624 2021-07-26T07:21:52.752 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:21:52.754 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:21:52.754 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:21:52.754 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:21:52.754 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:21:52.755 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:21:52.755 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:21:52.755 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:21:52.755 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:21:52.755 INFO:tasks.workunit.client.0.smithi071.stderr:460042: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:21:52.756 INFO:tasks.workunit.client.0.smithi071.stderr:460042: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:21:52.756 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:21:52.756 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:21:52.756 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:21:52.756 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:21:52.757 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:21:52.757 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:21:52.757 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:21:52.757 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:21:52.757 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:21:52.758 INFO:tasks.workunit.client.0.smithi071.stderr:460042: 1 2021-07-26T07:21:52.758 INFO:tasks.workunit.client.0.smithi071.stderr:460042: 2 2021-07-26T07:21:52.758 INFO:tasks.workunit.client.0.smithi071.stderr:460042: 3' 2021-07-26T07:21:52.758 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:21:52.759 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:52.759 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:21:52.759 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888066 2021-07-26T07:21:52.759 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888066 2021-07-26T07:21:52.759 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888066' 2021-07-26T07:21:52.760 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:52.760 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:21:52.760 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672972 2021-07-26T07:21:52.760 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672972 2021-07-26T07:21:52.761 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888066 1-42949672972' 2021-07-26T07:21:52.761 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:52.761 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: fl osd.2 up in weight 1 up_from 34 up_thru 0 down_at 30 last_clean_interval [15,29) [v2:127.0.0.1:6818/461276,v1:127.0.0.1:6819/461276] [v2:127.0.0.1:6820/461276,v1:127.0.0.1:6821/461276] exists,up e5d335fa-28d8-4dc2-b86b-256684f4ffec 2021-07-26T07:21:52.763 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:21:52.763 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:21:52.763 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:21:52.764 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: objectstore_tool: wait_for_clean 2021-07-26T07:21:52.765 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:21:52.765 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:21:52.765 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:21:52.766 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:21:52.766 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:21:52.766 INFO:tasks.workunit.client.0.smithi071.stderr:460045: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:21:52.766 INFO:tasks.workunit.client.0.smithi071.stderr:460045: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:21:52.767 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:21:52.767 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:21:52.767 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:21:52.767 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:21:52.768 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:21:52.768 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:21:52.768 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:21:52.768 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:21:52.768 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:21:52.769 INFO:tasks.workunit.client.0.smithi071.stderr:460045: 1 2021-07-26T07:21:52.769 INFO:tasks.workunit.client.0.smithi071.stderr:460045: 2 2021-07-26T07:21:52.769 INFO:tasks.workunit.client.0.smithi071.stderr:460045: 3' 2021-07-26T07:21:52.769 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:21:52.769 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:52.770 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:21:52.770 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888067 2021-07-26T07:21:52.770 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888067 2021-07-26T07:21:52.770 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888067' 2021-07-26T07:21:52.771 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:52.771 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:21:52.771 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672973 2021-07-26T07:21:52.771 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672973 2021-07-26T07:21:52.771 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888067 1-42949672973' 2021-07-26T07:21:52.772 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:54.942 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:21:54.943 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888067 2021-07-26T07:21:54.943 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888067 2021-07-26T07:21:54.943 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888068 1-42949672971 2-146028888067' 2021-07-26T07:21:54.943 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:54.943 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:21:54.945 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888066 2021-07-26T07:21:54.945 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888066 2021-07-26T07:21:54.945 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888068 1-42949672971 2-146028888067 3-146028888066' 2021-07-26T07:21:54.945 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:54.946 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-146028888068 2021-07-26T07:21:54.946 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:54.946 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:21:54.946 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-146028888068 2021-07-26T07:21:54.947 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:54.947 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888068 2021-07-26T07:21:54.947 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 146028888068' 2021-07-26T07:21:54.947 INFO:tasks.workunit.client.0.smithi071.stderr:460048: waiting osd.0 seq 146028888068 2021-07-26T07:21:54.947 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:21:54.948 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 146028888068 2021-07-26T07:21:54.948 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:21:54.948 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:21:54.948 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:21:54.948 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888068 -lt 146028888068 2021-07-26T07:21:54.949 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:54.949 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672971 2021-07-26T07:21:54.949 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:54.949 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:21:54.950 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672971 2021-07-26T07:21:54.950 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:54.950 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672971 2021-07-26T07:21:54.950 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2021-07-26T07:21:54.950 INFO:tasks.workunit.client.0.smithi071.stderr:460048: waiting osd.1 seq 42949672971 2021-07-26T07:21:54.951 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:21:54.951 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672974 -lt 42949672971 2021-07-26T07:21:55.077 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s ilush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:21:55.078 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888068 2021-07-26T07:21:55.078 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888068 2021-07-26T07:21:55.078 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888067 1-42949672973 2-146028888068' 2021-07-26T07:21:55.079 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:55.079 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:21:55.079 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888068 2021-07-26T07:21:55.079 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888068 2021-07-26T07:21:55.079 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888067 1-42949672973 2-146028888068 3-146028888068' 2021-07-26T07:21:55.080 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:55.080 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-146028888067 2021-07-26T07:21:55.080 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:55.080 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:21:55.081 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-146028888067 2021-07-26T07:21:55.081 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:55.081 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888067 2021-07-26T07:21:55.081 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2021-07-26T07:21:55.081 INFO:tasks.workunit.client.0.smithi071.stderr:460045: waiting osd.0 seq 146028888067 2021-07-26T07:21:55.082 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:21:55.082 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 146028888067 2021-07-26T07:21:55.082 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:21:55.082 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:21:55.083 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:21:55.083 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888068 -lt 146028888067 2021-07-26T07:21:55.083 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:55.083 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672973 2021-07-26T07:21:55.083 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:55.084 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:21:55.084 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672973 2021-07-26T07:21:55.084 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:55.084 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672973 2021-07-26T07:21:55.084 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2021-07-26T07:21:55.085 INFO:tasks.workunit.client.0.smithi071.stderr:460045: waiting osd.1 seq 42949672973 2021-07-26T07:21:55.085 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:21:55.085 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672974 -lt 42949672973 2021-07-26T07:21:55.098 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s iush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:21:55.098 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888066 2021-07-26T07:21:55.099 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888066 2021-07-26T07:21:55.099 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888066 1-42949672972 2-146028888066' 2021-07-26T07:21:55.099 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:55.099 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:21:55.100 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888067 2021-07-26T07:21:55.100 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888067 2021-07-26T07:21:55.100 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-146028888066 1-42949672972 2-146028888066 3-146028888067' 2021-07-26T07:21:55.100 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:55.101 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-146028888066 2021-07-26T07:21:55.101 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:55.101 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:21:55.101 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-146028888066 2021-07-26T07:21:55.101 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:55.102 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888066 2021-07-26T07:21:55.102 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2021-07-26T07:21:55.103 INFO:tasks.workunit.client.0.smithi071.stderr:460042: waiting osd.0 seq 146028888066 2021-07-26T07:21:55.103 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:21:55.104 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 146028888066 2021-07-26T07:21:55.104 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:21:55.105 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:21:55.105 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:21:55.106 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888068 -lt 146028888066 2021-07-26T07:21:55.106 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:55.106 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672972 2021-07-26T07:21:55.106 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:55.107 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:21:55.107 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672972 2021-07-26T07:21:55.107 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:55.107 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672972 2021-07-26T07:21:55.108 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2021-07-26T07:21:55.108 INFO:tasks.workunit.client.0.smithi071.stderr:460042: waiting osd.1 seq 42949672972 2021-07-26T07:21:55.108 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:21:55.108 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672974 -lt 42949672972 2021-07-26T07:21:56.241 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s inn $seqs 2021-07-26T07:21:56.241 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-146028888067 2021-07-26T07:21:56.241 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:56.242 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:21:56.242 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-146028888067 2021-07-26T07:21:56.243 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:56.243 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888067 2021-07-26T07:21:56.243 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2021-07-26T07:21:56.243 INFO:tasks.workunit.client.0.smithi071.stderr:460048: waiting osd.2 seq 146028888067 2021-07-26T07:21:56.244 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:21:56.244 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888068 -lt 146028888067 2021-07-26T07:21:56.244 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:56.244 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-146028888066 2021-07-26T07:21:56.245 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:56.245 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:21:56.245 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-146028888066 2021-07-26T07:21:56.245 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:56.246 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888066 2021-07-26T07:21:56.246 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 146028888066' 2021-07-26T07:21:56.246 INFO:tasks.workunit.client.0.smithi071.stderr:460048: waiting osd.3 seq 146028888066 2021-07-26T07:21:56.246 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:21:56.247 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888068 -lt 146028888066 2021-07-26T07:21:56.247 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:21:56.247 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:21:56.247 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:21:56.248 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:21:56.248 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:21:56.248 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:21:56.248 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:21:56.248 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:21:56.249 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:21:56.249 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:21:56.249 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:21:56.249 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:21:56.250 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:21:56.401 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/ceph $seqs 2021-07-26T07:21:56.402 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-146028888066 2021-07-26T07:21:56.402 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:56.402 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:21:56.402 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-146028888066 2021-07-26T07:21:56.403 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:56.403 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888066 2021-07-26T07:21:56.403 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 146028888066' 2021-07-26T07:21:56.403 INFO:tasks.workunit.client.0.smithi071.stderr:460042: waiting osd.2 seq 146028888066 2021-07-26T07:21:56.404 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:21:56.404 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888068 -lt 146028888066 2021-07-26T07:21:56.404 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:56.405 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-146028888067 2021-07-26T07:21:56.405 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:56.405 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:21:56.405 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-146028888067 2021-07-26T07:21:56.406 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:56.406 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888067 2021-07-26T07:21:56.406 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 146028888067' 2021-07-26T07:21:56.407 INFO:tasks.workunit.client.0.smithi071.stderr:460042: waiting osd.3 seq 146028888067 2021-07-26T07:21:56.407 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:21:56.407 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888068 -lt 146028888067 2021-07-26T07:21:56.407 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:21:56.407 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:21:56.408 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:21:56.408 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:21:56.408 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:21:56.408 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:21:56.408 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:21:56.410 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:21:56.411 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:21:56.411 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:21:56.411 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:21:56.411 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:21:56.412 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:21:56.412 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtn $seqs 2021-07-26T07:21:56.413 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-146028888068 2021-07-26T07:21:56.413 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:56.413 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:21:56.413 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-146028888068 2021-07-26T07:21:56.413 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:56.414 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888068 2021-07-26T07:21:56.414 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 146028888068' 2021-07-26T07:21:56.414 INFO:tasks.workunit.client.0.smithi071.stderr:460045: waiting osd.2 seq 146028888068 2021-07-26T07:21:56.414 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:21:56.415 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888068 -lt 146028888068 2021-07-26T07:21:56.415 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:56.415 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-146028888068 2021-07-26T07:21:56.415 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:56.415 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:21:56.416 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-146028888068 2021-07-26T07:21:56.416 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:56.416 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888068 2021-07-26T07:21:56.417 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 146028888068' 2021-07-26T07:21:56.417 INFO:tasks.workunit.client.0.smithi071.stderr:460045: waiting osd.3 seq 146028888068 2021-07-26T07:21:56.417 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:21:56.417 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888068 -lt 146028888068 2021-07-26T07:21:56.418 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:21:56.418 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:21:56.418 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:21:56.418 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:21:56.418 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:21:56.419 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:21:56.419 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:21:56.419 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:21:56.419 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:21:56.420 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:21:56.420 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:21:56.420 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:21:56.420 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:21:56.587 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:21:56.588 INFO:tasks.workunit.client.0.smithi071.stderr:460048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:21:56.588 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:21:56.588 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:21:56.588 INFO:tasks.workunit.client.0.smithi071.stderr:460048: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:21:56.589 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T07:21:56.771 INFO:tasks.workunit.client.0.smithi071.stderr:test/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:21:56.772 INFO:tasks.workunit.client.0.smithi071.stderr:460045: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:21:56.772 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:21:56.772 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:21:56.773 INFO:tasks.workunit.client.0.smithi071.stderr:460045: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:21:56.773 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T07:21:56.782 INFO:tasks.workunit.client.0.smithi071.stderr:est/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:21:56.782 INFO:tasks.workunit.client.0.smithi071.stderr:460042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:21:56.783 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:21:56.783 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:21:56.783 INFO:tasks.workunit.client.0.smithi071.stderr:460042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:21:56.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: run_in_background: return 0 2021-07-26T07:21:56.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T07:21:56.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 460040 2021-07-26T07:21:56.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: wait_background: for pid in $pids 2021-07-26T07:21:56.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: wait_background: wait 460044 2021-07-26T07:21:56.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: eval 'pids='\'''\''' 2021-07-26T07:21:56.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: wait_background: pids= 2021-07-26T07:21:56.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: wait_background: return 0 2021-07-26T07:21:56.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: unfound_erasure_coded: return_code=0 2021-07-26T07:21:56.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: unfound_erasure_coded: '[' 0 -ne 0 ']' 2021-07-26T07:21:56.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:896: unfound_erasure_coded: get_pg ecpool SOMETHING 2021-07-26T07:21:56.786 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=ecpool 2021-07-26T07:21:56.786 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=SOMETHING 2021-07-26T07:21:56.786 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map ecpool SOMETHING 2021-07-26T07:21:56.787 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:21:57.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:896: unfound_erasure_coded: local pg=1.0 2021-07-26T07:21:57.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: repair 1.0 2021-07-26T07:21:57.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: repair: local pgid=1.0 2021-07-26T07:21:57.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: get_last_scrub_stamp 1.0 2021-07-26T07:21:57.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:21:57.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:21:57.054 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:21:57.055 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:21:57.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: repair: local last_scrub=2021-07-26T07:21:36.649860+0000 2021-07-26T07:21:57.347 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: repair: ceph pg repair 1.0 2021-07-26T07:21:57.599 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 1.0s0 on osd.1 to repair 2021-07-26T07:21:57.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: repair: wait_for_scrub 1.0 2021-07-26T07:21:36.649860+0000 2021-07-26T07:21:57.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.0 2021-07-26T07:21:57.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:21:36.649860+0000 2021-07-26T07:21:57.610 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:21:57.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:21:57.611 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:21:57.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:21:57.611 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:21:57.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:21:57.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:21:57.612 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:21:57.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:21:36.649860+0000 '>' 2021-07-26T07:21:36.649860+0000 2021-07-26T07:21:57.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:21:58.873 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:21:58.874 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:21:58.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:21:58.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:21:58.875 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:21:58.876 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:21:58.876 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:21:59.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:21:36.649860+0000 '>' 2021-07-26T07:21:36.649860+0000 2021-07-26T07:21:59.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:22:00.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:22:00.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:22:00.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:22:00.152 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:22:00.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:22:00.155 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:22:00.156 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:22:00.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:21:36.649860+0000 '>' 2021-07-26T07:21:36.649860+0000 2021-07-26T07:22:00.424 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:22:01.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:22:01.425 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:22:01.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:22:01.426 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:22:01.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:22:01.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:22:01.427 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:22:01.698 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:21:36.649860+0000 '>' 2021-07-26T07:21:36.649860+0000 2021-07-26T07:22:01.699 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:22:02.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:22:02.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:22:02.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:22:02.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:22:02.701 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:22:02.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:22:02.702 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:22:02.977 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:21:36.649860+0000 '>' 2021-07-26T07:21:36.649860+0000 2021-07-26T07:22:02.978 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:22:03.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:22:03.979 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:22:03.980 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:22:03.980 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:22:03.980 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:22:03.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:22:03.981 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:22:04.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:21:36.649860+0000 '>' 2021-07-26T07:21:36.649860+0000 2021-07-26T07:22:04.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:22:05.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:22:05.257 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:22:05.257 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:22:05.258 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:22:05.258 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:22:05.258 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:22:05.258 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:22:05.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:22:00.434299+0000 '>' 2021-07-26T07:21:36.649860+0000 2021-07-26T07:22:05.528 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:22:05.529 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:902: unfound_erasure_coded: seq 1 60 2021-07-26T07:22:05.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:902: unfound_erasure_coded: for f in `seq 1 60` 2021-07-26T07:22:05.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:903: unfound_erasure_coded: ceph -s 2021-07-26T07:22:05.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:903: unfound_erasure_coded: grep '1/1 objects unfound' 2021-07-26T07:22:05.854 INFO:tasks.workunit.client.0.smithi071.stdout: 1/1 objects unfound (100.000%) 2021-07-26T07:22:05.854 INFO:tasks.workunit.client.0.smithi071.stdout: 1/1 objects unfound (100.000%) 2021-07-26T07:22:05.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:903: unfound_erasure_coded: break 2021-07-26T07:22:05.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:906: unfound_erasure_coded: ceph -s 2021-07-26T07:22:05.855 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:906: unfound_erasure_coded: grep '4 up' 2021-07-26T07:22:06.207 INFO:tasks.workunit.client.0.smithi071.stdout: osd: 4 osds: 4 up (since 14s), 4 in (since 41s) 2021-07-26T07:22:06.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: unfound_erasure_coded: ceph -s 2021-07-26T07:22:06.208 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: unfound_erasure_coded: grep '4 in' 2021-07-26T07:22:06.530 INFO:tasks.workunit.client.0.smithi071.stdout: osd: 4 osds: 4 up (since 14s), 4 in (since 41s) 2021-07-26T07:22:06.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:908: unfound_erasure_coded: ceph -s 2021-07-26T07:22:06.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:908: unfound_erasure_coded: grep '1/1 objects unfound' 2021-07-26T07:22:06.898 INFO:tasks.workunit.client.0.smithi071.stdout: 1/1 objects unfound (100.000%) 2021-07-26T07:22:06.899 INFO:tasks.workunit.client.0.smithi071.stdout: 1/1 objects unfound (100.000%) 2021-07-26T07:22:06.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2021-07-26T07:22:06.900 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:22:06.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:22:06.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:22:06.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:22:06.901 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:22:06.901 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:22:06.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:22:06.902 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:22:07.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:22:07.009 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:22:07.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:22:07.010 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:22:07.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:22:07.011 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:22:07.012 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:22:07.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:22:07.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:22:07.013 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:22:07.014 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:22:07.014 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:22:07.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:22:07.015 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:22:07.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:22:07.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:22:07.024 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:22:07.024 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:22:07.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:22:07.026 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:22:07.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T07:22:07.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-scrub-repair 0 2021-07-26T07:22:07.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2021-07-26T07:22:07.027 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T07:22:07.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2021-07-26T07:22:07.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:22:07.028 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:22:07.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:22:07.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:22:07.029 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:22:07.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:22:07.030 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:22:07.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:22:07.031 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:22:07.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:22:07.031 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:22:07.032 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:22:07.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:22:07.032 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:22:07.033 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:22:07.033 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:22:07.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:22:07.034 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T07:22:07.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2021-07-26T07:22:07.035 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:22:07.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:22:07.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.128899 2021-07-26T07:22:07.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.128899 2021-07-26T07:22:07.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:22:07.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:22:07.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T07:22:07.039 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T07:22:07.040 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T07:22:07.075 INFO:tasks.workunit:Running workunit scrub/osd-scrub-snaps.sh... 2021-07-26T07:22:07.076 DEBUG:teuthology.orchestra.run.smithi071:workunit test scrub/osd-scrub-snaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh 2021-07-26T07:22:07.136 INFO:tasks.workunit.client.0.smithi071.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T07:22:07.145 INFO:tasks.workunit.client.0.smithi071.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T07:22:07.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-07-26T07:22:07.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-07-26T07:22:07.146 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T07:22:07.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T07:22:07.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T07:22:07.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T07:22:07.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T07:22:07.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T07:22:07.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-scrub-snaps 2021-07-26T07:22:07.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:27: run: local dir=td/osd-scrub-snaps 2021-07-26T07:22:07.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:28: run: shift 2021-07-26T07:22:07.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: export CEPH_MON=127.0.0.1:7121 2021-07-26T07:22:07.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: CEPH_MON=127.0.0.1:7121 2021-07-26T07:22:07.149 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:31: run: export CEPH_ARGS 2021-07-26T07:22:07.149 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: uuidgen 2021-07-26T07:22:07.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: CEPH_ARGS+='--fsid=bb48d421-d23f-4bc8-9bea-096a86dc301c --auth-supported=none ' 2021-07-26T07:22:07.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:33: run: CEPH_ARGS+='--mon-host=127.0.0.1:7121 ' 2021-07-26T07:22:07.150 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:35: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T07:22:07.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: set 2021-07-26T07:22:07.150 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T07:22:07.156 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: local 'funcs=TEST_scrub_snaps 2021-07-26T07:22:07.156 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_scrub_snaps_primary 2021-07-26T07:22:07.156 INFO:tasks.workunit.client.0.smithi071.stderr:TEST_scrub_snaps_replica' 2021-07-26T07:22:07.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2021-07-26T07:22:07.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2021-07-26T07:22:07.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2021-07-26T07:22:07.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2021-07-26T07:22:07.157 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2021-07-26T07:22:07.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:22:07.158 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2021-07-26T07:22:07.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:22:07.158 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:22:07.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:22:07.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:22:07.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:22:07.159 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:22:07.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:22:07.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:22:07.160 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:22:07.160 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:22:07.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:22:07.161 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:22:07.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:22:07.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:22:07.161 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:22:07.162 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:22:07.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:22:07.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:22:07.162 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2021-07-26T07:22:07.162 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:22:07.163 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:22:07.163 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:22:07.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.464690 2021-07-26T07:22:07.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:22:07.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:22:07.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2021-07-26T07:22:07.166 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:22:07.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:22:07.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:22:07.167 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.464690 2021-07-26T07:22:07.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:22:07.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T07:22:07.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T07:22:07.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:22:07.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2021-07-26T07:22:07.168 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps td/osd-scrub-snaps 2021-07-26T07:22:07.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:171: TEST_scrub_snaps: local dir=td/osd-scrub-snaps 2021-07-26T07:22:07.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:172: TEST_scrub_snaps: local poolname=test 2021-07-26T07:22:07.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:173: TEST_scrub_snaps: local OBJS=16 2021-07-26T07:22:07.169 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:174: TEST_scrub_snaps: local OSDS=1 2021-07-26T07:22:07.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:176: TEST_scrub_snaps: TESTDATA=testdata.464690 2021-07-26T07:22:07.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:178: TEST_scrub_snaps: run_mon td/osd-scrub-snaps a --osd_pool_default_size=1 2021-07-26T07:22:07.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2021-07-26T07:22:07.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:22:07.170 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:22:07.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:22:07.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2021-07-26T07:22:07.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=1 2021-07-26T07:22:07.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:22:07.511 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:22:07.512 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:22:07.512 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:22:07.512 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:22:07.512 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:22:07.513 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.464690/$cluster-$name.asok' 2021-07-26T07:22:07.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 2021-07-26T07:22:07.754 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:22:07.756 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:22:07.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:22:07.757 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:22:07.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:22:07.758 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:22:07.758 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:22:07.758 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:22:07.758 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:22:07.759 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:22:07.759 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:22:07.759 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:22:07.759 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.464690/ceph-mon.a.asok 2021-07-26T07:22:07.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:22:07.760 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.464690/ceph-mon.a.asok config get fsid 2021-07-26T07:22:07.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:22:07.854 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:22:07.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:22:07.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:22:07.855 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:22:07.855 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:22:07.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:22:07.856 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:22:07.856 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:22:07.857 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:22:07.857 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:22:07.857 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.464690/ceph-mon.a.asok 2021-07-26T07:22:07.857 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:22:07.858 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.464690/ceph-mon.a.asok config get mon_host 2021-07-26T07:22:07.948 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:179: TEST_scrub_snaps: run_mgr td/osd-scrub-snaps x 2021-07-26T07:22:07.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2021-07-26T07:22:07.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:22:07.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:22:07.949 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:22:07.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2021-07-26T07:22:07.950 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:22:08.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:22:08.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:22:08.783 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:22:08.784 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:22:08.784 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:22:08.784 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:22:08.784 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.464690/$cluster-$name.asok' 2021-07-26T07:22:08.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:22:08.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:22:08.806 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: expr 1 - 1 2021-07-26T07:22:08.806 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: seq 0 0 2021-07-26T07:22:08.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:22:08.812 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:182: TEST_scrub_snaps: run_osd td/osd-scrub-snaps 0 2021-07-26T07:22:08.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2021-07-26T07:22:08.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:22:08.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:22:08.813 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:22:08.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2021-07-26T07:22:08.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=bb48d421-d23f-4bc8-9bea-096a86dc301c --auth-supported=none --mon-host=127.0.0.1:7121 ' 2021-07-26T07:22:08.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:22:08.814 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:22:08.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:22:08.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2021-07-26T07:22:08.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2021-07-26T07:22:08.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:22:08.815 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:22:08.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:22:08.816 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2021-07-26T07:22:08.816 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:22:08.816 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:22:08.817 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:22:08.817 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:22:08.817 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:22:08.817 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:22:08.818 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.464690/$cluster-$name.asok' 2021-07-26T07:22:08.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' 2021-07-26T07:22:08.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:22:08.818 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:22:08.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:22:08.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2021-07-26T07:22:08.819 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2021-07-26T07:22:08.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:22:08.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:22:08.820 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:22:08.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:22:08.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:22:08.821 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2021-07-26T07:22:08.822 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:22:08.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cdfe1bce-ff28-4eed-b5c6-9ed52393f15f 2021-07-26T07:22:08.822 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cdfe1bce-ff28-4eed-b5c6-9ed52393f15f' 2021-07-26T07:22:08.823 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 cdfe1bce-ff28-4eed-b5c6-9ed52393f15f 2021-07-26T07:22:08.824 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:22:08.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCgYv5gHR4PMRAAzP6qW9KeuWjaHHmApjkBiQ== 2021-07-26T07:22:08.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCgYv5gHR4PMRAAzP6qW9KeuWjaHHmApjkBiQ=="}' 2021-07-26T07:22:08.825 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cdfe1bce-ff28-4eed-b5c6-9ed52393f15f -i td/osd-scrub-snaps/0/new.json 2021-07-26T07:22:09.072 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:22:09.082 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2021-07-26T07:22:09.083 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=bb48d421-d23f-4bc8-9bea-096a86dc301c --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCgYv5gHR4PMRAAzP6qW9KeuWjaHHmApjkBiQ== --osd-uuid cdfe1bce-ff28-4eed-b5c6-9ed52393f15f 2021-07-26T07:22:09.105 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:22:09.105+0000 7f6b0b39d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:22:09.108 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:22:09.106+0000 7f6b0b39d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:22:09.108 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:22:09.107+0000 7f6b0b39d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:22:09.109 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:22:09.108+0000 7f6b0b39d0c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2021-07-26T07:22:09.109 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:22:09.108+0000 7f6b0b39d0c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2021-07-26T07:22:09.109 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:22:09.108+0000 7f6b0b39d0c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2021-07-26T07:22:09.189 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:22:09.188+0000 7f6b0b39d0c0 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2021-07-26T07:22:14.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2021-07-26T07:22:14.848 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:22:14.849 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:22:14.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:22:14.849 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:22:15.185 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:22:15.186 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:22:15.186 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=bb48d421-d23f-4bc8-9bea-096a86dc301c --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:22:15.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:22:15.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:22:15.187 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:22:15.204 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:22:15.204+0000 7fa5316820c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:22:15.206 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:22:15.205+0000 7fa5316820c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:22:15.207 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:22:15.206+0000 7fa5316820c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:22:15.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:22:15.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:22:15.455 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:22:15.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:22:15.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:22:15.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:22:15.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:22:15.457 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:22:15.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:22:15.458 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:22:15.726 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:22:15.725+0000 7fa5316820c0 -1 Falling back to public interface 2021-07-26T07:22:15.732 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:22:16.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:22:16.733 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:22:16.734 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:22:16.734 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:22:16.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:22:16.735 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:22:17.006 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:22:18.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:22:18.008 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:22:18.009 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:22:18.009 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:22:18.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:22:18.010 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:22:18.139 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:22:18.138+0000 7fa5316820c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:22:18.321 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:22:19.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:22:19.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:22:19.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:22:19.325 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:22:19.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:22:19.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:22:19.596 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/465811,v1:127.0.0.1:6803/465811] [v2:127.0.0.1:6804/465811,v1:127.0.0.1:6805/465811] exists,up cdfe1bce-ff28-4eed-b5c6-9ed52393f15f 2021-07-26T07:22:19.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:22:19.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:22:19.597 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:22:19.598 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:186: TEST_scrub_snaps: ceph osd set noscrub 2021-07-26T07:22:20.028 INFO:tasks.workunit.client.0.smithi071.stderr:noscrub is set 2021-07-26T07:22:20.038 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:187: TEST_scrub_snaps: ceph osd set nodeep-scrub 2021-07-26T07:22:20.531 INFO:tasks.workunit.client.0.smithi071.stderr:nodeep-scrub is set 2021-07-26T07:22:20.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:190: TEST_scrub_snaps: create_pool test 1 1 2021-07-26T07:22:20.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-07-26T07:22:20.924 INFO:tasks.workunit.client.0.smithi071.stderr:pool 'test' created 2021-07-26T07:22:20.934 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:22:21.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:191: TEST_scrub_snaps: wait_for_clean 2021-07-26T07:22:21.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:22:21.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:22:21.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:22:21.936 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:22:21.937 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:22:21.938 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:22:21.938 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:22:21.938 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:22:21.938 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:22:21.939 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:22:22.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:22:22.035 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:22:22.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:22:22.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:22:22.036 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:22:22.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:22:22.299 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:22:22.300 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:22:22.300 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:22:22.456 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836482 2021-07-26T07:22:22.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836482 2021-07-26T07:22:22.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836482' 2021-07-26T07:22:22.457 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:22:22.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836482 2021-07-26T07:22:22.457 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:22:22.462 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:22:22.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836482 2021-07-26T07:22:22.462 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:22:22.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836482 2021-07-26T07:22:22.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836482' 2021-07-26T07:22:22.465 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 21474836482 2021-07-26T07:22:22.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:22:22.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836481 -lt 21474836482 2021-07-26T07:22:22.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:22:23.722 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:22:23.723 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:22:23.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836482 -lt 21474836482 2021-07-26T07:22:23.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:22:23.998 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:23.999 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:24.332 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:22:24.333 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:24.333 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:24.333 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:24.333 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:24.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:24.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:24.334 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:24.633 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:22:24.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:24.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:24.634 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:24.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:22:24.971 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:22:24.972 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:22:24.972 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: ceph osd dump 2021-07-26T07:22:24.973 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: grep '^pool.*['\'']test['\'']' 2021-07-26T07:22:24.973 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: awk '{ print $2 }' 2021-07-26T07:22:25.233 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: poolid=1 2021-07-26T07:22:25.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:194: TEST_scrub_snaps: dd if=/dev/urandom of=testdata.464690 bs=1032 count=1 2021-07-26T07:22:25.235 INFO:tasks.workunit.client.0.smithi071.stderr:1+0 records in 2021-07-26T07:22:25.235 INFO:tasks.workunit.client.0.smithi071.stderr:1+0 records out 2021-07-26T07:22:25.235 INFO:tasks.workunit.client.0.smithi071.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.6561e-05 s, 22.2 MB/s 2021-07-26T07:22:25.236 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: seq 1 16 2021-07-26T07:22:25.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:25.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj1 testdata.464690 2021-07-26T07:22:25.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:25.349 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj2 testdata.464690 2021-07-26T07:22:25.466 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:25.467 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj3 testdata.464690 2021-07-26T07:22:25.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:25.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj4 testdata.464690 2021-07-26T07:22:25.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:25.651 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj5 testdata.464690 2021-07-26T07:22:25.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:25.709 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj6 testdata.464690 2021-07-26T07:22:25.768 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:25.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj7 testdata.464690 2021-07-26T07:22:25.826 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:25.827 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj8 testdata.464690 2021-07-26T07:22:25.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:25.886 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj9 testdata.464690 2021-07-26T07:22:25.969 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:25.970 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj10 testdata.464690 2021-07-26T07:22:26.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:26.028 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj11 testdata.464690 2021-07-26T07:22:26.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:26.086 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj12 testdata.464690 2021-07-26T07:22:26.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:26.145 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj13 testdata.464690 2021-07-26T07:22:26.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:26.220 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj14 testdata.464690 2021-07-26T07:22:26.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:26.295 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj15 testdata.464690 2021-07-26T07:22:26.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2021-07-26T07:22:26.371 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj16 testdata.464690 2021-07-26T07:22:26.446 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: get_primary test obj1 2021-07-26T07:22:26.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:22:26.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:22:26.447 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:22:26.448 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:22:26.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: local primary=0 2021-07-26T07:22:26.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:202: TEST_scrub_snaps: create_scenario td/osd-scrub-snaps test testdata.464690 0 2021-07-26T07:22:26.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2021-07-26T07:22:26.711 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2021-07-26T07:22:26.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.464690 2021-07-26T07:22:26.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2021-07-26T07:22:26.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2021-07-26T07:22:26.712 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2021-07-26T07:22:26.913 INFO:tasks.workunit.client.0.smithi071.stdout:created pool test snap snap1 2021-07-26T07:22:26.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.464690 bs=256 count=1 2021-07-26T07:22:26.915 INFO:tasks.workunit.client.0.smithi071.stderr:1+0 records in 2021-07-26T07:22:26.916 INFO:tasks.workunit.client.0.smithi071.stderr:1+0 records out 2021-07-26T07:22:26.916 INFO:tasks.workunit.client.0.smithi071.stderr:256 bytes copied, 5.8842e-05 s, 4.4 MB/s 2021-07-26T07:22:26.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.464690 2021-07-26T07:22:27.005 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.464690 2021-07-26T07:22:27.081 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.464690 2021-07-26T07:22:27.131 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2021-07-26T07:22:27.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2021-07-26T07:22:27.132 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.464690 2021-07-26T07:22:27.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2021-07-26T07:22:27.215 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.464690 2021-07-26T07:22:27.290 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2021-07-26T07:22:27.291 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.464690 2021-07-26T07:22:27.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2021-07-26T07:22:27.366 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.464690 2021-07-26T07:22:27.441 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2021-07-26T07:22:27.442 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.464690 2021-07-26T07:22:27.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2021-07-26T07:22:27.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.464690 2021-07-26T07:22:27.625 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2021-07-26T07:22:27.626 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.464690 2021-07-26T07:22:27.700 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2021-07-26T07:22:27.701 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.464690 2021-07-26T07:22:27.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2021-07-26T07:22:27.835 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.464690 2021-07-26T07:22:28.043 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2021-07-26T07:22:28.044 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2021-07-26T07:22:28.293 INFO:tasks.workunit.client.0.smithi071.stdout:created pool test snap snap2 2021-07-26T07:22:28.294 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.464690 bs=256 count=2 2021-07-26T07:22:28.295 INFO:tasks.workunit.client.0.smithi071.stderr:2+0 records in 2021-07-26T07:22:28.295 INFO:tasks.workunit.client.0.smithi071.stderr:2+0 records out 2021-07-26T07:22:28.296 INFO:tasks.workunit.client.0.smithi071.stderr:512 bytes copied, 5.9971e-05 s, 8.5 MB/s 2021-07-26T07:22:28.296 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.464690 2021-07-26T07:22:28.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2021-07-26T07:22:28.412 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2021-07-26T07:22:28.621 INFO:tasks.workunit.client.0.smithi071.stdout:created pool test snap snap3 2021-07-26T07:22:28.622 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.464690 bs=256 count=3 2021-07-26T07:22:28.623 INFO:tasks.workunit.client.0.smithi071.stderr:3+0 records in 2021-07-26T07:22:28.624 INFO:tasks.workunit.client.0.smithi071.stderr:3+0 records out 2021-07-26T07:22:28.624 INFO:tasks.workunit.client.0.smithi071.stderr:768 bytes copied, 7.9516e-05 s, 9.7 MB/s 2021-07-26T07:22:28.624 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.464690 2021-07-26T07:22:28.747 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2021-07-26T07:22:28.748 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2021-07-26T07:22:28.940 INFO:tasks.workunit.client.0.smithi071.stdout:created pool test snap snap4 2021-07-26T07:22:28.941 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.464690 bs=256 count=4 2021-07-26T07:22:28.942 INFO:tasks.workunit.client.0.smithi071.stderr:4+0 records in 2021-07-26T07:22:28.943 INFO:tasks.workunit.client.0.smithi071.stderr:4+0 records out 2021-07-26T07:22:28.943 INFO:tasks.workunit.client.0.smithi071.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 7.3418e-05 s, 13.9 MB/s 2021-07-26T07:22:28.943 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.464690 2021-07-26T07:22:29.066 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.464690 2021-07-26T07:22:29.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2021-07-26T07:22:29.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2021-07-26T07:22:29.333 INFO:tasks.workunit.client.0.smithi071.stdout:created pool test snap snap5 2021-07-26T07:22:29.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2021-07-26T07:22:29.335 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2021-07-26T07:22:29.634 INFO:tasks.workunit.client.0.smithi071.stdout:created pool test snap snap6 2021-07-26T07:22:29.635 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.464690 bs=256 count=6 2021-07-26T07:22:29.636 INFO:tasks.workunit.client.0.smithi071.stderr:6+0 records in 2021-07-26T07:22:29.637 INFO:tasks.workunit.client.0.smithi071.stderr:6+0 records out 2021-07-26T07:22:29.637 INFO:tasks.workunit.client.0.smithi071.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 8.3306e-05 s, 18.4 MB/s 2021-07-26T07:22:29.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.464690 2021-07-26T07:22:29.769 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2021-07-26T07:22:29.770 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2021-07-26T07:22:30.045 INFO:tasks.workunit.client.0.smithi071.stdout:created pool test snap snap7 2021-07-26T07:22:30.047 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2021-07-26T07:22:30.188 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2021-07-26T07:22:30.298 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2021-07-26T07:22:30.374 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2021-07-26T07:22:30.375 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:22:30.376 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:22:30.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:22:30.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:22:30.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:22:30.478 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:22:30.479 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2021-07-26T07:22:31.762 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:22:32.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:22:32.030 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2021-07-26T07:22:32.672 INFO:tasks.workunit.client.0.smithi071.stdout:WARNING: only removing head with clones present 2021-07-26T07:22:32.673 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:ff7b1f36:::obj1:head# 2021-07-26T07:22:33.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2021-07-26T07:22:33.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2021-07-26T07:22:34.009 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:22:34.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:22:34.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2021-07-26T07:22:35.203 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:c52c9666:::obj5:2# 2021-07-26T07:22:35.725 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2021-07-26T07:22:35.726 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2021-07-26T07:22:36.466 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:22:36.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:22:36.983 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:22:36.984 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2021-07-26T07:22:38.021 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2021-07-26T07:22:38.023 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:38.023 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2021-07-26T07:22:38.023 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:38.024 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2021-07-26T07:22:38.024 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2021-07-26T07:22:38.025 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2021-07-26T07:22:38.025 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2021-07-26T07:22:38.026 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2021-07-26T07:22:38.026 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2021-07-26T07:22:38.027 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2021-07-26T07:22:38.027 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:38.027 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2021-07-26T07:22:38.028 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2021-07-26T07:22:38.028 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2021-07-26T07:22:38.029 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2021-07-26T07:22:38.029 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:38.030 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2021-07-26T07:22:38.031 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:38.031 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:38.031 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:38.031 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2021-07-26T07:22:38.032 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:38.032 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2021-07-26T07:22:38.032 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:38.032 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:38.032 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:38.033 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2021-07-26T07:22:38.033 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:38.033 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:38.033 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:38.034 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2021-07-26T07:22:38.034 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2021-07-26T07:22:38.034 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:38.034 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:38.034 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2021-07-26T07:22:38.035 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2021-07-26T07:22:38.035 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:38.035 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:38.035 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2021-07-26T07:22:38.036 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2021-07-26T07:22:38.036 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2021-07-26T07:22:38.037 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2021-07-26T07:22:38.695 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:c52c9666:::obj5:1# 2021-07-26T07:22:39.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2021-07-26T07:22:40.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2021-07-26T07:22:40.251 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:40.251 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2021-07-26T07:22:40.252 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:40.253 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2021-07-26T07:22:40.253 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2021-07-26T07:22:40.253 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2021-07-26T07:22:40.254 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2021-07-26T07:22:40.254 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2021-07-26T07:22:40.254 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2021-07-26T07:22:40.254 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2021-07-26T07:22:40.255 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:40.255 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2021-07-26T07:22:40.255 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2021-07-26T07:22:40.255 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2021-07-26T07:22:40.255 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2021-07-26T07:22:40.256 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:40.256 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2021-07-26T07:22:40.256 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:40.256 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:40.257 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:40.257 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2021-07-26T07:22:40.257 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:40.258 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2021-07-26T07:22:40.258 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:40.258 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:40.258 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:40.259 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2021-07-26T07:22:40.259 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:40.259 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:40.259 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:40.259 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2021-07-26T07:22:40.260 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2021-07-26T07:22:40.260 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:40.260 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:40.260 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2021-07-26T07:22:40.261 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2021-07-26T07:22:40.261 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2021-07-26T07:22:40.261 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:40.261 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2021-07-26T07:22:40.262 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2021-07-26T07:22:40.262 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2021-07-26T07:22:40.263 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2021-07-26T07:22:40.263 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2021-07-26T07:22:40.264 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2021-07-26T07:22:40.908 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:22:41.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:22:41.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.464690 bs=256 count=18 2021-07-26T07:22:41.180 INFO:tasks.workunit.client.0.smithi071.stderr:18+0 records in 2021-07-26T07:22:41.180 INFO:tasks.workunit.client.0.smithi071.stderr:18+0 records out 2021-07-26T07:22:41.181 INFO:tasks.workunit.client.0.smithi071.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000146617 s, 31.4 MB/s 2021-07-26T07:22:41.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.464690 2021-07-26T07:22:42.470 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2021-07-26T07:22:43.271 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:22:43.543 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:22:43.544 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.464690 bs=256 count=15 2021-07-26T07:22:43.545 INFO:tasks.workunit.client.0.smithi071.stderr:15+0 records in 2021-07-26T07:22:43.545 INFO:tasks.workunit.client.0.smithi071.stderr:15+0 records out 2021-07-26T07:22:43.545 INFO:tasks.workunit.client.0.smithi071.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000142784 s, 26.9 MB/s 2021-07-26T07:22:43.545 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.464690 2021-07-26T07:22:44.835 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2021-07-26T07:22:44.836 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2021-07-26T07:22:45.681 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:22:46.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:22:46.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2021-07-26T07:22:46.865 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:0ee9ae15:::obj4:7# 2021-07-26T07:22:47.382 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2021-07-26T07:22:48.671 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2021-07-26T07:22:48.672 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2021-07-26T07:22:48.672 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.672 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2021-07-26T07:22:48.672 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2021-07-26T07:22:48.673 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2021-07-26T07:22:48.673 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2021-07-26T07:22:48.673 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2021-07-26T07:22:48.673 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2021-07-26T07:22:48.674 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2021-07-26T07:22:48.674 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:48.674 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2021-07-26T07:22:48.674 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2021-07-26T07:22:48.675 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2021-07-26T07:22:48.675 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2021-07-26T07:22:48.675 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2021-07-26T07:22:48.676 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.676 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:48.676 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2021-07-26T07:22:48.676 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.677 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2021-07-26T07:22:48.677 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:48.677 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.677 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2021-07-26T07:22:48.677 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:48.678 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.678 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2021-07-26T07:22:48.678 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2021-07-26T07:22:48.678 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.679 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2021-07-26T07:22:48.679 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2021-07-26T07:22:48.679 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.679 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2021-07-26T07:22:48.680 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2021-07-26T07:22:48.680 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.680 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.681 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.681 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.681 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.681 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.682 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2021-07-26T07:22:48.682 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2021-07-26T07:22:48.682 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2021-07-26T07:22:49.328 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:22:49.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:22:49.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2021-07-26T07:22:50.272 INFO:tasks.workunit.client.0.smithi071.stdout:remove #1:461f8b5e:::obj16:7# 2021-07-26T07:22:50.789 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2021-07-26T07:22:52.077 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2021-07-26T07:22:52.078 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2021-07-26T07:22:52.078 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2021-07-26T07:22:52.078 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2021-07-26T07:22:52.078 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2021-07-26T07:22:52.079 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2021-07-26T07:22:52.079 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2021-07-26T07:22:52.079 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2021-07-26T07:22:52.079 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2021-07-26T07:22:52.080 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:52.080 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2021-07-26T07:22:52.080 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2021-07-26T07:22:52.080 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2021-07-26T07:22:52.080 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2021-07-26T07:22:52.081 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2021-07-26T07:22:52.081 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:52.081 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2021-07-26T07:22:52.081 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2021-07-26T07:22:52.082 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:52.082 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2021-07-26T07:22:52.082 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2021-07-26T07:22:52.082 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2021-07-26T07:22:52.082 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2021-07-26T07:22:52.083 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2021-07-26T07:22:52.083 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2021-07-26T07:22:52.083 INFO:tasks.workunit.client.0.smithi071.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2021-07-26T07:22:52.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2021-07-26T07:22:52.084 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2021-07-26T07:22:52.084 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2021-07-26T07:22:52.728 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:22:52.998 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:22:52.999 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2021-07-26T07:22:54.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:22:54.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2021-07-26T07:22:54.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:22:54.196 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.464690 bs=256 count=7 2021-07-26T07:22:54.196 INFO:tasks.workunit.client.0.smithi071.stderr:7+0 records in 2021-07-26T07:22:54.197 INFO:tasks.workunit.client.0.smithi071.stderr:7+0 records out 2021-07-26T07:22:54.197 INFO:tasks.workunit.client.0.smithi071.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 9.5986e-05 s, 18.7 MB/s 2021-07-26T07:22:54.197 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.464690 2021-07-26T07:22:55.474 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2021-07-26T07:22:56.320 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:22:56.320 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2021-07-26T07:22:56.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:22:56.838 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2021-07-26T07:22:58.018 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2021-07-26T07:22:58.843 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:22:58.843 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2021-07-26T07:22:59.359 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:22:59.360 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2021-07-26T07:23:00.621 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2021-07-26T07:23:01.320 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:23:01.320 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2021-07-26T07:23:01.836 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:23:01.837 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2021-07-26T07:23:03.002 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2021-07-26T07:23:03.770 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:23:03.771 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2021-07-26T07:23:04.045 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:23:04.046 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2021-07-26T07:23:05.226 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2021-07-26T07:23:06.017 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:23:06.017 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2021-07-26T07:23:06.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:23:06.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2021-07-26T07:23:07.712 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2021-07-26T07:23:08.489 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:23:08.489 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2021-07-26T07:23:08.764 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:23:08.765 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2021-07-26T07:23:09.920 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2021-07-26T07:23:10.644 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:23:10.644 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2021-07-26T07:23:10.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:23:10.914 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2021-07-26T07:23:12.124 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2021-07-26T07:23:12.881 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:23:12.882 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2021-07-26T07:23:13.147 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:23:13.148 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2021-07-26T07:23:14.506 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2021-07-26T07:23:15.236 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:23:15.237 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2021-07-26T07:23:15.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:23:15.758 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2021-07-26T07:23:16.898 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2021-07-26T07:23:16.899 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2021-07-26T07:23:17.652 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2021-07-26T07:23:17.652 INFO:tasks.workunit.client.0.smithi071.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2021-07-26T07:23:17.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2021-07-26T07:23:17.923 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2021-07-26T07:23:19.163 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2021-07-26T07:23:19.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2021-07-26T07:23:19.164 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:204: TEST_scrub_snaps: rm -f testdata.464690 2021-07-26T07:23:19.165 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: expr 1 - 1 2021-07-26T07:23:19.167 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: seq 0 0 2021-07-26T07:23:19.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:23:19.171 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:208: TEST_scrub_snaps: activate_osd td/osd-scrub-snaps 0 2021-07-26T07:23:19.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-scrub-snaps 2021-07-26T07:23:19.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:23:19.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:23:19.172 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:23:19.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-scrub-snaps/0 2021-07-26T07:23:19.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=bb48d421-d23f-4bc8-9bea-096a86dc301c --auth-supported=none --mon-host=127.0.0.1:7121 ' 2021-07-26T07:23:19.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:23:19.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:23:19.173 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:23:19.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2021-07-26T07:23:19.174 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2021-07-26T07:23:19.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:23:19.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:23:19.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:23:19.175 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2021-07-26T07:23:19.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:23:19.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:23:19.176 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:23:19.176 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:23:19.176 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:19.177 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:23:19.177 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.464690/$cluster-$name.asok' 2021-07-26T07:23:19.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' 2021-07-26T07:23:19.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:23:19.177 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2021-07-26T07:23:19.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2021-07-26T07:23:19.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:23:19.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:23:19.178 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:23:19.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:23:19.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:23:19.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-scrub-snaps/0 2021-07-26T07:23:19.179 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:23:19.180 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:23:19.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=bb48d421-d23f-4bc8-9bea-096a86dc301c --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:23:19.181 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-scrub-snaps/0/whoami 2021-07-26T07:23:19.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:23:19.181 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:23:19.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:23:19.182 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:23:19.193 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:19.193+0000 7f3050b6c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:19.195 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:19.195+0000 7f3050b6c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:19.196 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:19.196+0000 7f3050b6c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:19.448 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:23:19.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:23:19.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:23:19.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:23:19.449 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:23:19.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:19.450 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:23:19.450 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:23:19.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:19.451 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:23:19.713 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 9 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/465811,v1:127.0.0.1:6803/465811] [v2:127.0.0.1:6804/465811,v1:127.0.0.1:6805/465811] exists,up cdfe1bce-ff28-4eed-b5c6-9ed52393f15f 2021-07-26T07:23:19.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:23:19.716 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:23:19.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:23:19.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:211: TEST_scrub_snaps: wait_for_clean 2021-07-26T07:23:19.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:23:19.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:23:19.717 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:23:19.718 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:23:19.718 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:23:19.718 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:23:19.718 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:23:19.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:23:19.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:23:19.719 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:23:19.723 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:19.722+0000 7f3050b6c0c0 -1 Falling back to public interface 2021-07-26T07:23:19.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:23:19.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:23:19.807 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:23:19.808 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:23:19.808 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:23:20.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids=0 2021-07-26T07:23:20.071 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:23:20.072 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:23:20.072 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:23:22.110 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:22.108+0000 7f3050b6c0c0 -1 osd.0 18 log_to_monitors {default=true} 2021-07-26T07:23:25.191 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=90194313218 2021-07-26T07:23:25.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 90194313218 2021-07-26T07:23:25.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-90194313218' 2021-07-26T07:23:25.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:23:25.192 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-90194313218 2021-07-26T07:23:25.193 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:23:25.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:23:25.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-90194313218 2021-07-26T07:23:25.194 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:23:25.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=90194313218 2021-07-26T07:23:25.195 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 90194313218' 2021-07-26T07:23:25.195 INFO:tasks.workunit.client.0.smithi071.stdout:waiting osd.0 seq 90194313218 2021-07-26T07:23:25.196 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:23:25.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 90194313218 2021-07-26T07:23:25.463 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:23:26.464 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:23:26.465 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:23:26.739 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 90194313218 -lt 90194313218 2021-07-26T07:23:26.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:23:26.739 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:23:26.740 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:23:27.104 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:23:27.105 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:23:27.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:23:27.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:23:27.105 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:23:27.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:23:27.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:23:27.106 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:23:27.363 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:23:27.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:23:27.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:23:27.364 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:23:27.691 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:23:27.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:23:27.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:23:27.692 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:213: TEST_scrub_snaps: local pgid=1.0 2021-07-26T07:23:27.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:214: TEST_scrub_snaps: pg_scrub 1.0 2021-07-26T07:23:27.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: pg_scrub: local pgid=1.0 2021-07-26T07:23:27.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: pg_scrub: get_last_scrub_stamp 1.0 2021-07-26T07:23:27.693 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:23:27.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:23:27.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:23:27.694 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:23:27.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: pg_scrub: local last_scrub=2021-07-26T07:22:20.848707+0000 2021-07-26T07:23:27.992 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: pg_scrub: ceph pg scrub 1.0 2021-07-26T07:23:28.242 INFO:tasks.workunit.client.0.smithi071.stderr:instructing pg 1.0 on osd.0 to scrub 2021-07-26T07:23:28.252 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: pg_scrub: wait_for_scrub 1.0 2021-07-26T07:22:20.848707+0000 2021-07-26T07:23:28.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: wait_for_scrub: local pgid=1.0 2021-07-26T07:23:28.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: wait_for_scrub: local last_scrub=2021-07-26T07:22:20.848707+0000 2021-07-26T07:23:28.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: wait_for_scrub: local sname=last_scrub_stamp 2021-07-26T07:23:28.253 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i=0 )) 2021-07-26T07:23:28.254 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:23:28.254 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:23:28.254 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:23:28.254 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:23:28.255 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:23:28.255 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:23:28.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:22:20.848707+0000 '>' 2021-07-26T07:22:20.848707+0000 2021-07-26T07:23:28.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:23:29.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:23:29.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:23:29.514 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:23:29.514 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:23:29.514 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:23:29.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:23:29.515 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:23:29.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:22:20.848707+0000 '>' 2021-07-26T07:22:20.848707+0000 2021-07-26T07:23:29.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:23:30.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:23:30.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:23:30.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:23:30.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:23:30.785 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:23:30.786 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:23:30.786 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:23:31.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:22:20.848707+0000 '>' 2021-07-26T07:22:20.848707+0000 2021-07-26T07:23:31.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:23:32.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:23:32.057 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:23:32.058 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:23:32.058 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:23:32.058 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:23:32.061 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:23:32.061 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:23:32.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:22:20.848707+0000 '>' 2021-07-26T07:22:20.848707+0000 2021-07-26T07:23:32.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: wait_for_scrub: sleep 1 2021-07-26T07:23:33.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i++ )) 2021-07-26T07:23:33.326 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: wait_for_scrub: (( i < 300 )) 2021-07-26T07:23:33.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2021-07-26T07:23:33.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: get_last_scrub_stamp: local pgid=1.0 2021-07-26T07:23:33.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: get_last_scrub_stamp: local sname=last_scrub_stamp 2021-07-26T07:23:33.327 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1470: get_last_scrub_stamp: ceph --format json pg dump pgs 2021-07-26T07:23:33.328 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1471: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2021-07-26T07:23:33.599 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: wait_for_scrub: test 2021-07-26T07:23:29.188855+0000 '>' 2021-07-26T07:22:20.848707+0000 2021-07-26T07:23:33.600 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: wait_for_scrub: return 0 2021-07-26T07:23:33.600 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:33.601 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: wc -l 2021-07-26T07:23:33.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: test 2 = 2 2021-07-26T07:23:33.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:220: TEST_scrub_snaps: rados list-inconsistent-pg test 2021-07-26T07:23:33.633 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:222: TEST_scrub_snaps: jq '. | length' td/osd-scrub-snaps/json 2021-07-26T07:23:33.634 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:222: TEST_scrub_snaps: test 1 = 1 2021-07-26T07:23:33.635 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:224: TEST_scrub_snaps: jq -r '.[0]' td/osd-scrub-snaps/json 2021-07-26T07:23:33.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:224: TEST_scrub_snaps: test 1.0 = 1.0 2021-07-26T07:23:33.637 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:226: TEST_scrub_snaps: rados list-inconsistent-obj 1.0 2021-07-26T07:23:33.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:231: TEST_scrub_snaps: jq .inconsistents 2021-07-26T07:23:33.659 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:231: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2021-07-26T07:23:33.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:238: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2021-07-26T07:23:33.682 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:238: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2021-07-26T07:23:33.705 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:239: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2021-07-26T07:23:33.706 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2021-07-26T07:23:33.710 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:241: TEST_scrub_snaps: rados list-inconsistent-snapset 1.0 2021-07-26T07:23:33.727 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: jq .inconsistents 2021-07-26T07:23:33.728 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2021-07-26T07:23:33.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:622: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2021-07-26T07:23:33.752 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:622: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2021-07-26T07:23:33.779 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:623: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2021-07-26T07:23:33.780 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2021-07-26T07:23:33.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:624: TEST_scrub_snaps: test no = yes 2021-07-26T07:23:33.781 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:629: TEST_scrub_snaps: test '' = yes 2021-07-26T07:23:33.781 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: find td/osd-scrub-snaps 2021-07-26T07:23:33.782 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: grep 'osd[^/]*\.pid' 2021-07-26T07:23:33.782 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: pidfiles=td/osd-scrub-snaps/osd.0.pid 2021-07-26T07:23:33.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:635: TEST_scrub_snaps: pids= 2021-07-26T07:23:33.783 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:636: TEST_scrub_snaps: for pidfile in ${pidfiles} 2021-07-26T07:23:33.784 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:638: TEST_scrub_snaps: cat td/osd-scrub-snaps/osd.0.pid 2021-07-26T07:23:33.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:638: TEST_scrub_snaps: pids+='483409 ' 2021-07-26T07:23:33.784 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:641: TEST_scrub_snaps: ERRORS=0 2021-07-26T07:23:33.784 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: seq 1 7 2021-07-26T07:23:33.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2021-07-26T07:23:33.785 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap1 2021-07-26T07:23:33.984 INFO:tasks.workunit.client.0.smithi071.stdout:removed pool test snap snap1 2021-07-26T07:23:33.985 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2021-07-26T07:23:33.986 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap2 2021-07-26T07:23:34.285 INFO:tasks.workunit.client.0.smithi071.stdout:removed pool test snap snap2 2021-07-26T07:23:34.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2021-07-26T07:23:34.287 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap3 2021-07-26T07:23:34.503 INFO:tasks.workunit.client.0.smithi071.stdout:removed pool test snap snap3 2021-07-26T07:23:34.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2021-07-26T07:23:34.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap4 2021-07-26T07:23:34.713 INFO:tasks.workunit.client.0.smithi071.stdout:removed pool test snap snap4 2021-07-26T07:23:34.714 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2021-07-26T07:23:34.715 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap5 2021-07-26T07:23:34.914 INFO:tasks.workunit.client.0.smithi071.stdout:removed pool test snap snap5 2021-07-26T07:23:34.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2021-07-26T07:23:34.916 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap6 2021-07-26T07:23:35.116 INFO:tasks.workunit.client.0.smithi071.stdout:removed pool test snap snap6 2021-07-26T07:23:35.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2021-07-26T07:23:35.117 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap7 2021-07-26T07:23:35.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed pool test snap snap7 2021-07-26T07:23:35.336 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:647: TEST_scrub_snaps: sleep 5 2021-07-26T07:23:40.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:648: TEST_scrub_snaps: local -i loop=0 2021-07-26T07:23:40.337 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:649: TEST_scrub_snaps: ceph pg dump pgs 2021-07-26T07:23:40.338 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:649: TEST_scrub_snaps: grep -q snaptrim 2021-07-26T07:23:40.599 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T07:23:40.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:651: TEST_scrub_snaps: ceph pg dump pgs 2021-07-26T07:23:40.609 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:651: TEST_scrub_snaps: grep -q snaptrim_error 2021-07-26T07:23:40.859 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T07:23:40.868 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: sleep 2 2021-07-26T07:23:42.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:656: TEST_scrub_snaps: loop+=1 2021-07-26T07:23:42.870 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: (( 1 >= 10 )) 2021-07-26T07:23:42.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:649: TEST_scrub_snaps: ceph pg dump pgs 2021-07-26T07:23:42.871 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:649: TEST_scrub_snaps: grep -q snaptrim 2021-07-26T07:23:43.130 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T07:23:43.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:651: TEST_scrub_snaps: ceph pg dump pgs 2021-07-26T07:23:43.139 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:651: TEST_scrub_snaps: grep -q snaptrim_error 2021-07-26T07:23:43.386 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T07:23:43.395 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: sleep 2 2021-07-26T07:23:45.396 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:656: TEST_scrub_snaps: loop+=1 2021-07-26T07:23:45.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: (( 2 >= 10 )) 2021-07-26T07:23:45.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:649: TEST_scrub_snaps: ceph pg dump pgs 2021-07-26T07:23:45.397 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:649: TEST_scrub_snaps: grep -q snaptrim 2021-07-26T07:23:45.667 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T07:23:45.677 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:651: TEST_scrub_snaps: ceph pg dump pgs 2021-07-26T07:23:45.678 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:651: TEST_scrub_snaps: grep -q snaptrim_error 2021-07-26T07:23:45.926 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T07:23:45.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: break 2021-07-26T07:23:45.935 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:662: TEST_scrub_snaps: ceph pg dump pgs 2021-07-26T07:23:46.181 INFO:tasks.workunit.client.0.smithi071.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T07:23:46.181 INFO:tasks.workunit.client.0.smithi071.stdout:1.0 35 0 0 0 0 23416 0 0 60 60 active+clean+inconsistent+snaptrim_error 2021-07-26T07:23:39.185738+0000 23'60 29:78 [0] 0 [0] 0 18'56 2021-07-26T07:23:29.188855+0000 0'0 2021-07-26T07:22:20.848707+0000 7 2021-07-26T07:23:46.181 INFO:tasks.workunit.client.0.smithi071.stdout: 2021-07-26T07:23:46.181 INFO:tasks.workunit.client.0.smithi071.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T07:23:46.182 INFO:tasks.workunit.client.0.smithi071.stderr:dumped pgs 2021-07-26T07:23:46.192 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:664: TEST_scrub_snaps: for pid in $pids 2021-07-26T07:23:46.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:666: TEST_scrub_snaps: kill -0 483409 2021-07-26T07:23:46.193 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:673: TEST_scrub_snaps: kill_daemons td/osd-scrub-snaps 2021-07-26T07:23:46.193 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:23:46.193 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:23:46.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:23:46.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:23:46.194 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:23:46.503 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:23:46.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:675: TEST_scrub_snaps: declare -a err_strings 2021-07-26T07:23:46.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:676: TEST_scrub_snaps: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' 2021-07-26T07:23:46.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:677: TEST_scrub_snaps: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' 2021-07-26T07:23:46.504 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:678: TEST_scrub_snaps: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' 2021-07-26T07:23:46.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:679: TEST_scrub_snaps: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' 2021-07-26T07:23:46.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:680: TEST_scrub_snaps: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' 2021-07-26T07:23:46.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:681: TEST_scrub_snaps: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' 2021-07-26T07:23:46.505 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:682: TEST_scrub_snaps: err_strings[6]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' 2021-07-26T07:23:46.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:683: TEST_scrub_snaps: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' 2021-07-26T07:23:46.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:684: TEST_scrub_snaps: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' 2021-07-26T07:23:46.506 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:685: TEST_scrub_snaps: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' 2021-07-26T07:23:46.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:686: TEST_scrub_snaps: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' 2021-07-26T07:23:46.507 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:687: TEST_scrub_snaps: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' 2021-07-26T07:23:46.508 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:688: TEST_scrub_snaps: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' 2021-07-26T07:23:46.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:689: TEST_scrub_snaps: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' 2021-07-26T07:23:46.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:690: TEST_scrub_snaps: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' 2021-07-26T07:23:46.509 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:691: TEST_scrub_snaps: err_strings[15]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' 2021-07-26T07:23:46.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:692: TEST_scrub_snaps: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' 2021-07-26T07:23:46.510 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:693: TEST_scrub_snaps: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' 2021-07-26T07:23:46.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:694: TEST_scrub_snaps: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' 2021-07-26T07:23:46.511 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:695: TEST_scrub_snaps: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' 2021-07-26T07:23:46.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:696: TEST_scrub_snaps: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2021-07-26T07:23:46.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:697: TEST_scrub_snaps: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' 2021-07-26T07:23:46.512 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:698: TEST_scrub_snaps: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: 1,2,3,4,5,6,7 was r -2...repaired' 2021-07-26T07:23:46.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.513 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.514 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.515 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.519 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.520 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.521 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.522 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.523 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.524 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.525 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.526 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.530 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.531 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.532 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.533 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.534 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.535 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.537 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.538 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.539 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.540 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2021-07-26T07:23:46.541 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: 1,2,3,4,5,6,7 was r -2...repaired' td/osd-scrub-snaps/osd.0.log 2021-07-26T07:23:46.542 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:709: TEST_scrub_snaps: '[' 0 '!=' 0 ']' 2021-07-26T07:23:46.543 INFO:tasks.workunit.client.0.smithi071.stdout:TEST PASSED 2021-07-26T07:23:46.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:715: TEST_scrub_snaps: echo 'TEST PASSED' 2021-07-26T07:23:46.548 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:716: TEST_scrub_snaps: return 0 2021-07-26T07:23:46.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2021-07-26T07:23:46.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2021-07-26T07:23:46.549 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:23:46.550 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2021-07-26T07:23:46.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:23:46.550 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:23:46.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:23:46.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:23:46.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:23:46.551 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:23:46.552 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:23:46.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:23:46.552 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:23:46.552 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:23:46.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:23:46.553 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:23:46.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:23:46.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:23:46.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:23:46.554 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:23:46.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:23:46.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:23:46.555 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2021-07-26T07:23:46.558 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:23:46.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:46.559 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:23:46.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.464690 2021-07-26T07:23:46.559 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:23:46.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:23:46.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2021-07-26T07:23:46.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2021-07-26T07:23:46.560 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2021-07-26T07:23:46.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2021-07-26T07:23:46.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2021-07-26T07:23:46.561 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:23:46.562 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2021-07-26T07:23:46.562 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:23:46.562 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:23:46.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:23:46.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:23:46.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:23:46.563 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:23:46.566 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:23:46.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:23:46.567 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:23:46.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:23:46.567 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:23:46.568 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:23:46.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:23:46.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:23:46.568 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:23:46.569 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:23:46.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:23:46.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:23:46.569 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2021-07-26T07:23:46.569 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:23:46.570 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:46.570 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:23:46.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.464690 2021-07-26T07:23:46.570 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:23:46.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:23:46.571 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2021-07-26T07:23:46.574 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:23:46.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:46.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:23:46.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.464690 2021-07-26T07:23:46.575 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:23:46.575 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:23:46.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:23:46.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2021-07-26T07:23:46.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_primary td/osd-scrub-snaps 2021-07-26T07:23:46.576 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1160: TEST_scrub_snaps_primary: local dir=td/osd-scrub-snaps 2021-07-26T07:23:46.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1161: TEST_scrub_snaps_primary: ORIG_ARGS='--fsid=bb48d421-d23f-4bc8-9bea-096a86dc301c --auth-supported=none --mon-host=127.0.0.1:7121 ' 2021-07-26T07:23:46.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: TEST_scrub_snaps_primary: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3' 2021-07-26T07:23:46.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: TEST_scrub_snaps_primary: _scrub_snaps_multi td/osd-scrub-snaps primary 2021-07-26T07:23:46.577 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:720: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2021-07-26T07:23:46.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:721: _scrub_snaps_multi: local poolname=test 2021-07-26T07:23:46.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:722: _scrub_snaps_multi: local OBJS=16 2021-07-26T07:23:46.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:723: _scrub_snaps_multi: local OSDS=2 2021-07-26T07:23:46.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:724: _scrub_snaps_multi: local which=primary 2021-07-26T07:23:46.578 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:726: _scrub_snaps_multi: TESTDATA=testdata.464690 2021-07-26T07:23:46.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:728: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2021-07-26T07:23:46.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2021-07-26T07:23:46.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:23:46.579 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:23:46.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:23:46.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2021-07-26T07:23:46.580 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2021-07-26T07:23:46.931 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:23:46.932 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:23:46.932 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:23:46.932 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:23:46.932 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:46.933 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:23:46.933 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.464690/$cluster-$name.asok' 2021-07-26T07:23:46.933 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2021-07-26T07:23:47.184 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:23:47.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:23:47.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:23:47.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:23:47.185 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:23:47.186 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:23:47.186 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:23:47.186 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:23:47.186 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:23:47.187 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:23:47.187 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:47.187 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:23:47.188 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.464690/ceph-mon.a.asok 2021-07-26T07:23:47.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:23:47.188 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.464690/ceph-mon.a.asok config get fsid 2021-07-26T07:23:47.282 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:23:47.282 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:23:47.283 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:23:47.283 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:23:47.283 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:23:47.283 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:23:47.283 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:23:47.284 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:23:47.284 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:23:47.284 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:47.284 INFO:tasks.workunit.client.0.smithi071.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:23:47.285 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.464690/ceph-mon.a.asok 2021-07-26T07:23:47.285 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:23:47.285 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.464690/ceph-mon.a.asok config get mon_host 2021-07-26T07:23:47.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:729: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2021-07-26T07:23:47.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2021-07-26T07:23:47.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:23:47.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:23:47.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:23:47.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2021-07-26T07:23:47.379 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:23:48.203 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:23:48.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:23:48.204 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:23:48.204 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:23:48.204 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:48.205 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:23:48.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.464690/$cluster-$name.asok' 2021-07-26T07:23:48.205 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:23:48.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:23:48.228 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: expr 2 - 1 2021-07-26T07:23:48.229 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: seq 0 1 2021-07-26T07:23:48.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:23:48.234 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2021-07-26T07:23:48.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2021-07-26T07:23:48.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:23:48.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:23:48.235 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:23:48.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2021-07-26T07:23:48.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=bb48d421-d23f-4bc8-9bea-096a86dc301c --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3' 2021-07-26T07:23:48.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:23:48.236 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:23:48.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:23:48.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2021-07-26T07:23:48.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2021-07-26T07:23:48.237 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:23:48.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:23:48.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:23:48.238 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2021-07-26T07:23:48.238 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:23:48.239 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:23:48.239 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:23:48.239 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:23:48.239 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:48.239 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:23:48.240 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.464690/$cluster-$name.asok' 2021-07-26T07:23:48.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' 2021-07-26T07:23:48.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:23:48.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:23:48.240 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:23:48.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2021-07-26T07:23:48.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2021-07-26T07:23:48.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:23:48.241 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:23:48.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:23:48.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:23:48.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:23:48.242 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2021-07-26T07:23:48.242 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:23:48.243 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7d86043f-389e-4885-9148-aeec6902c5dc 2021-07-26T07:23:48.243 INFO:tasks.workunit.client.0.smithi071.stdout:add osd0 7d86043f-389e-4885-9148-aeec6902c5dc 2021-07-26T07:23:48.244 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7d86043f-389e-4885-9148-aeec6902c5dc' 2021-07-26T07:23:48.244 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:23:48.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAEY/5gKhzSDhAAn9K3u8bJGFphImDip+CeZA== 2021-07-26T07:23:48.249 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAEY/5gKhzSDhAAn9K3u8bJGFphImDip+CeZA=="}' 2021-07-26T07:23:48.250 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7d86043f-389e-4885-9148-aeec6902c5dc -i td/osd-scrub-snaps/0/new.json 2021-07-26T07:23:48.542 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:23:48.553 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2021-07-26T07:23:48.554 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=bb48d421-d23f-4bc8-9bea-096a86dc301c --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAEY/5gKhzSDhAAn9K3u8bJGFphImDip+CeZA== --osd-uuid 7d86043f-389e-4885-9148-aeec6902c5dc 2021-07-26T07:23:48.580 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:48.579+0000 7ff8dd2400c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:48.582 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:48.581+0000 7ff8dd2400c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:48.582 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:48.582+0000 7ff8dd2400c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:48.583 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:48.582+0000 7ff8dd2400c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2021-07-26T07:23:48.583 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:48.582+0000 7ff8dd2400c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2021-07-26T07:23:48.584 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:48.582+0000 7ff8dd2400c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2021-07-26T07:23:48.659 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:48.658+0000 7ff8dd2400c0 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2021-07-26T07:23:54.376 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2021-07-26T07:23:54.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:23:54.377 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:23:54.378 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd0 key to auth repository 2021-07-26T07:23:54.378 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:23:54.719 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:23:54.719 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.0 2021-07-26T07:23:54.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=bb48d421-d23f-4bc8-9bea-096a86dc301c --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:23:54.720 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:23:54.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:23:54.721 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:23:54.743 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:54.742+0000 7fd5729580c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:54.744 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:54.744+0000 7fd5729580c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:54.745 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:54.745+0000 7fd5729580c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:55.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:23:55.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:23:55.053 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:23:55.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:23:55.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:23:55.054 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:55.055 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:23:55.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:23:55.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:55.056 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:23:55.261 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:55.260+0000 7fd5729580c0 -1 Falling back to public interface 2021-07-26T07:23:55.322 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:23:56.323 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:23:56.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:56.324 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:23:56.325 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:23:56.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:56.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:23:56.602 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:23:57.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:23:57.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:57.604 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:23:57.605 INFO:tasks.workunit.client.0.smithi071.stdout:2 2021-07-26T07:23:57.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:57.605 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:23:57.644 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:57.643+0000 7fd5729580c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:23:57.918 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:23:58.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:23:58.920 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:58.921 INFO:tasks.workunit.client.0.smithi071.stdout:3 2021-07-26T07:23:58.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:23:58.921 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:58.922 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:23:59.197 INFO:tasks.workunit.client.0.smithi071.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/485912,v1:127.0.0.1:6803/485912] [v2:127.0.0.1:6804/485912,v1:127.0.0.1:6805/485912] exists,up 7d86043f-389e-4885-9148-aeec6902c5dc 2021-07-26T07:23:59.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:23:59.202 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:23:59.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:23:59.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:23:59.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2021-07-26T07:23:59.203 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2021-07-26T07:23:59.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:23:59.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:23:59.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:23:59.204 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/1 2021-07-26T07:23:59.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=bb48d421-d23f-4bc8-9bea-096a86dc301c --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3' 2021-07-26T07:23:59.205 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:23:59.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:23:59.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:23:59.206 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2021-07-26T07:23:59.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2021-07-26T07:23:59.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:23:59.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:23:59.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:23:59.207 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2021-07-26T07:23:59.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:23:59.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:23:59.208 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:23:59.209 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:23:59.209 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:59.209 INFO:tasks.workunit.client.0.smithi071.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.464690 2021-07-26T07:23:59.209 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.464690/$cluster-$name.asok' 2021-07-26T07:23:59.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' 2021-07-26T07:23:59.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:23:59.210 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:23:59.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:23:59.211 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2021-07-26T07:23:59.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2021-07-26T07:23:59.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:23:59.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:23:59.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:23:59.212 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:23:59.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:23:59.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/1 2021-07-26T07:23:59.213 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:23:59.213 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8732e56a-83b0-4956-ab1f-5fd5fd20aaea 2021-07-26T07:23:59.214 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8732e56a-83b0-4956-ab1f-5fd5fd20aaea' 2021-07-26T07:23:59.214 INFO:tasks.workunit.client.0.smithi071.stdout:add osd1 8732e56a-83b0-4956-ab1f-5fd5fd20aaea 2021-07-26T07:23:59.215 INFO:tasks.workunit.client.0.smithi071.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:23:59.216 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPY/5gcKTgDBAAQKqCx0mMNdM3wgFsk2MM1w== 2021-07-26T07:23:59.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPY/5gcKTgDBAAQKqCx0mMNdM3wgFsk2MM1w=="}' 2021-07-26T07:23:59.217 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8732e56a-83b0-4956-ab1f-5fd5fd20aaea -i td/osd-scrub-snaps/1/new.json 2021-07-26T07:23:59.573 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:23:59.584 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/1/new.json 2021-07-26T07:23:59.586 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=bb48d421-d23f-4bc8-9bea-096a86dc301c --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAPY/5gcKTgDBAAQKqCx0mMNdM3wgFsk2MM1w== --osd-uuid 8732e56a-83b0-4956-ab1f-5fd5fd20aaea 2021-07-26T07:23:59.611 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:59.611+0000 7f7c954aa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:59.613 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:59.612+0000 7f7c954aa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:59.614 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:59.613+0000 7f7c954aa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:59.614 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:59.613+0000 7f7c954aa0c0 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2021-07-26T07:23:59.615 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:59.613+0000 7f7c954aa0c0 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2021-07-26T07:23:59.615 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:59.613+0000 7f7c954aa0c0 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2021-07-26T07:23:59.678 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:23:59.678+0000 7f7c954aa0c0 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2021-07-26T07:24:05.350 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2021-07-26T07:24:05.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:24:05.351 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:24:05.352 INFO:tasks.workunit.client.0.smithi071.stdout:adding osd1 key to auth repository 2021-07-26T07:24:05.352 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:24:05.693 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:24:05.694 INFO:tasks.workunit.client.0.smithi071.stdout:start osd.1 2021-07-26T07:24:05.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:24:05.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=bb48d421-d23f-4bc8-9bea-096a86dc301c --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.464690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:24:05.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:24:05.695 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:24:05.717 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:24:05.716+0000 7f37a2b010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:24:05.719 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:24:05.718+0000 7f37a2b010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:24:05.719 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:24:05.719+0000 7f37a2b010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:24:06.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:24:06.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:24:06.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:24:06.016 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:24:06.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:24:06.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:24:06.017 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:24:06.018 INFO:tasks.workunit.client.0.smithi071.stdout:0 2021-07-26T07:24:06.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:24:06.018 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:24:06.236 INFO:tasks.workunit.client.0.smithi071.stderr:2021-07-26T07:24:06.236+0000 7f37a2b010c0 -1 Falling back to public interface 2021-07-26T07:24:06.325 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:24:07.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:24:07.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:24:07.327 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:24:07.328 INFO:tasks.workunit.client.0.smithi071.stdout:1 2021-07-26T07:24:07.328 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:24:07.329 INFO:tasks.workunit.client.0.smithi071.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:24:07.519 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-07-26T07:24:07.520 DEBUG:teuthology.task.console_log:Killing console logger for smithi071 2021-07-26T07:24:07.522 DEBUG:teuthology.exit:Finished running handlers